Skip to content

pytorch mini batch size

조회 수 40 추천 수 0 2019.12.19 07:12:55

https://stackoverflow.com/questions/52518324/how-to-compensate-if-i-cant-do-a-large-batch-size-in-neural-network/52523847

 

 

4

In pytorch, when you perform the backward step (calling loss.backward() or similar) the gradients are accumulated in-place. This means that if you call loss.backward() multiple times, the previously calculated gradients are not replaced, but in stead the new gradients get added on to the previous ones. That is why, when using pytorch, it is usually necessary to explicitly zero the gradients between minibatches (by calling optimiser.zero_grad() or similar).

If your batch size is limited, you can simulate a larger batch size by breaking a large batch up into smaller pieces, and only calling optimiser.step() to update the model parameters after all the pieces have been processed.

For example, suppose you are only able to do batches of size 64, but you wish to simulate a batch size of 128. If the original training loop looks like:

optimiser.zero_grad()
loss = model(batch_data) # batch_data is a batch of size 128
loss.backward()
optimiser.step()

then you could change this to:

optimiser.zero_grad()

smaller_batches = batch_data[:64], batch_data[64:128]
for batch in smaller_batches:
    loss = model(batch) / 2
    loss.backward()

optimiser.step()

and the updates to the model parameters would be the same in each case (apart maybe from some small numerical error). Note that you have to rescale the loss to make the update the same.

List of Articles
번호 제목 글쓴이 날짜 조회 수
1618 retinanet nvidia [3] WHRIA 2020-01-12 22
1617 lvm [2] WHRIA 2020-01-09 11
1616 usb 3.1 + DP WHRIA 2020-01-07 12
1615 raid 6 rebuild WHRIA 2020-01-07 11
1614 add extra raid disk WHRIA 2020-01-05 11
1613 참고 또 참고 WHRIA 2020-01-04 32
1612 R graph WHRIA 2019-12-29 16
1611 melanoma awareness [1] WHRIA 2019-12-27 15
1610 윈도우 raid ahci 전환 WHRIA 2019-12-26 11
1609 lvm 확장 [1] WHRIA 2019-12-25 18
1608 pytorch object detect / retinanet WHRIA 2019-12-22 171
1607 kaggle leakage WHRIA 2019-12-19 10
1606 pytorch - caffe WHRIA 2019-12-19 11
» pytorch mini batch size WHRIA 2019-12-19 40
1604 windows softraid monitor WHRIA 2019-12-19 17

Powered by Xpress Engine / Designed by Sketchbook

sketchbook5, 스케치북5

sketchbook5, 스케치북5

나눔글꼴 설치 안내


이 PC에는 나눔글꼴이 설치되어 있지 않습니다.

이 사이트를 나눔글꼴로 보기 위해서는
나눔글꼴을 설치해야 합니다.

설치 취소