Skip to content

pytorch mini batch size

2019.12.19 07:12

WHRIA 조회 수:82

https://stackoverflow.com/questions/52518324/how-to-compensate-if-i-cant-do-a-large-batch-size-in-neural-network/52523847

 

 

4

In pytorch, when you perform the backward step (calling loss.backward() or similar) the gradients are accumulated in-place. This means that if you call loss.backward() multiple times, the previously calculated gradients are not replaced, but in stead the new gradients get added on to the previous ones. That is why, when using pytorch, it is usually necessary to explicitly zero the gradients between minibatches (by calling optimiser.zero_grad() or similar).

If your batch size is limited, you can simulate a larger batch size by breaking a large batch up into smaller pieces, and only calling optimiser.step() to update the model parameters after all the pieces have been processed.

For example, suppose you are only able to do batches of size 64, but you wish to simulate a batch size of 128. If the original training loop looks like:

optimiser.zero_grad()
loss = model(batch_data) # batch_data is a batch of size 128
loss.backward()
optimiser.step()

then you could change this to:

optimiser.zero_grad()

smaller_batches = batch_data[:64], batch_data[64:128]
for batch in smaller_batches:
    loss = model(batch) / 2
    loss.backward()

optimiser.step()

and the updates to the model parameters would be the same in each case (apart maybe from some small numerical error). Note that you have to rescale the loss to make the update the same.

번호 제목 글쓴이 날짜 조회 수
1611 melanoma awareness [1] WHRIA 2019.12.27 45
1610 윈도우 raid ahci 전환 WHRIA 2019.12.26 37
1609 lvm 확장 [1] WHRIA 2019.12.25 48
1608 pytorch object detect / retinanet WHRIA 2019.12.22 214
1607 kaggle leakage WHRIA 2019.12.19 35
1606 pytorch - caffe WHRIA 2019.12.19 37
» pytorch mini batch size WHRIA 2019.12.19 82
1604 windows softraid monitor WHRIA 2019.12.19 66
1603 raid monitor WHRIA 2019.12.15 31
1602 SAS controller SAS 9212-4i WHRIA 2019.12.08 143
1601 Basic tutorial to develop driver on windows WHRIA 2019.12.07 375
1600 Keratinocytic Skin Cancer Detection on the Face Using Region-Based Convolutional Neural Network WHRIA 2019.12.07 324
1599 linux raid WHRIA 2019.11.30 46
1598 Micro- and macro-averages WHRIA 2019.11.29 140
1597 mcnemar WHRIA 2019.11.20 76

Powered by Xpress Engine / Designed by Sketchbook

sketchbook5, 스케치북5

sketchbook5, 스케치북5

나눔글꼴 설치 안내


이 PC에는 나눔글꼴이 설치되어 있지 않습니다.

이 사이트를 나눔글꼴로 보기 위해서는
나눔글꼴을 설치해야 합니다.

설치 취소