### Describe the bug

I am trying to get CLIP to fine-tuning with my code.
When I tried to run it on multiple GPUs using accelerate, I encountered the following phenomenon.

- Validation dataloader stuck in 2nd epoch only on multi-GPU
Specifically, when the "for inputs in valid_loader:" process is finished, it does not proceed to the next step. train_loader process is completed. Also, both train and valid are working correctly in the first epoch.
The accelerate command at that time is as follows.
`accelerate launch --multi_gpu --num_processes=2 {script_name.py} {--arg1} {--arg2} ...`

- This will not happen when single GPU is used.
`CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...`
- Setting num_workers=0 in dataloader did not change the result.

### Steps to reproduce the bug

1. The codes for fine-tuning the regular CLIP were updated for accelerate.
2. Run the code with the accelerate command as `accelerate launch --multi_gpu --num_processes=2 {script_name.py} {--arg1} {--arg2} ...` and the above problem will occur.
3. CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...` , it works fine.

### Expected behavior

It Should end normally as if it was run on a single GPU.

### Environment info

Since `datasets-cli env` did not work, the environment is described below.

- OS: Ubuntu 22.04 with Docker
- Docker: 24.0.5, build ced0996
- Python: 3.10.12
- torch==2.0.1
- accelerate==0.21.0
- transformers==4.33.1
Dataloader stuck on multiple GPUs
What type of dataset are you using in this script? `torch.utils.data.Dataset` or `datasets.Dataset`? Please share the `datasets` package version if it's the latter. Otherwise, it's better to move this issue to the `accelerate` repo. Very sorry, I thought I had a repo in `accelerate!`
I will close this issue and repo the issue in the appropriate place.