<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="bogoychev-etal-2018-accelerating">
    <titleInfo>
        <title>Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Nikolay</namePart>
        <namePart type="family">Bogoychev</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kenneth</namePart>
        <namePart type="family">Heafield</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alham</namePart>
        <namePart type="given">Fikri</namePart>
        <namePart type="family">Aji</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marcin</namePart>
        <namePart type="family">Junczys-Dowmunt</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2018-oct-nov</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Ellen</namePart>
            <namePart type="family">Riloff</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">David</namePart>
            <namePart type="family">Chiang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Julia</namePart>
            <namePart type="family">Hockenmaier</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jun’ichi</namePart>
            <namePart type="family">Tsujii</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Brussels, Belgium</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In order to extract the best possible performance from asynchronous stochastic gradient descent one must increase the mini-batch size and scale the learning rate accordingly. In order to achieve further speedup we introduce a technique that delays gradient updates effectively increasing the mini-batch size. Unfortunately with the increase of mini-batch size we worsen the stale gradient problem in asynchronous stochastic gradient descent (SGD) which makes the model convergence poor. We introduce local optimizers which mitigate the stale gradient problem and together with fine tuning our momentum we are able to train a shallow machine translation system 27% faster than an optimized baseline with negligible penalty in BLEU.</abstract>
    <identifier type="citekey">bogoychev-etal-2018-accelerating</identifier>
    <identifier type="doi">10.18653/v1/D18-1332</identifier>
    <location>
        <url>https://aclanthology.org/D18-1332</url>
    </location>
    <part>
        <date>2018-oct-nov</date>
        <extent unit="page">
            <start>2991</start>
            <end>2996</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation
%A Bogoychev, Nikolay
%A Heafield, Kenneth
%A Aji, Alham Fikri
%A Junczys-Dowmunt, Marcin
%Y Riloff, Ellen
%Y Chiang, David
%Y Hockenmaier, Julia
%Y Tsujii, Jun’ichi
%S Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
%D 2018
%8 oct nov
%I Association for Computational Linguistics
%C Brussels, Belgium
%F bogoychev-etal-2018-accelerating
%X In order to extract the best possible performance from asynchronous stochastic gradient descent one must increase the mini-batch size and scale the learning rate accordingly. In order to achieve further speedup we introduce a technique that delays gradient updates effectively increasing the mini-batch size. Unfortunately with the increase of mini-batch size we worsen the stale gradient problem in asynchronous stochastic gradient descent (SGD) which makes the model convergence poor. We introduce local optimizers which mitigate the stale gradient problem and together with fine tuning our momentum we are able to train a shallow machine translation system 27% faster than an optimized baseline with negligible penalty in BLEU.
%R 10.18653/v1/D18-1332
%U https://aclanthology.org/D18-1332
%U https://doi.org/10.18653/v1/D18-1332
%P 2991-2996

Markdown (Informal)

[Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation](https://aclanthology.org/D18-1332) (Bogoychev et al., EMNLP 2018)

• Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation (Bogoychev et al., EMNLP 2018)

ACL

• Nikolay Bogoychev, Kenneth Heafield, Alham Fikri Aji, and Marcin Junczys-Dowmunt. 2018. Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2991–2996, Brussels, Belgium. Association for Computational Linguistics.