Papers of the day   All papers

Energy and Policy Considerations for Deep Learning in NLP

Comments

Eric Topol: Perhaps the least acknowledged downside of deep neural net #AI models: the carbon footprint But this key preprint is starting to get noticed https://arxiv.org/abs/1906.02243 by @strubell @andrewmccallum https://jamanetwork.com/journals/jama/fullarticle/2758612 @AndrewLBeam https://www.technologyreview.com/s/613630/training-a-single-ai-model-can-emit-as-much-carbon-as-five-cars-in-their-lifetimes/ @techreview @_KarenHao https://t.co/j3dcGityEv

14 replies, 310 likes


Isaac Kohane: Why we should admire @fastdotai & @jeremyphoward focus on computational efficiency (eg superconvergence) https://www.theverge.com/2018/5/7/17316010/fast-ai-speed-test-stanford-dawnbench-google-intel

2 replies, 161 likes


Sylvain ❄️👨🏻‍🎓: Training a single machine-learning model can have over four times the footprint of a car over its entire lifetime 😶 https://arxiv.org/abs/1906.02243 https://t.co/kAdkhxITi7

14 replies, 119 likes


Lior Pachter: We (w/@sinabooeshaghi @VeigaBeltrame) computed the carbon footprint of running Cell Ranger vs. kallisto bustools for scRNA-seq. Turns out for one dataset it's the difference between driving a car from LA to Mexico vs. driving a few blocks in Pasadena. https://twitter.com/EricTopol/status/1216187183402373122

1 replies, 62 likes


Kyle McDonald: @toxi @golan @lennyjpg @pixlpa @sterlingcrispin assuming 1xGPU rig averages 200W. ethereum uses 11.04TWh/year and 29.17 kWh/transaction, bitcoin 636kWh/transaction. NAS is 656 MWh. sources: https://digiconomist.net/ethereum-energy-consumption https://arxiv.org/abs/1906.02243

1 replies, 45 likes


Arthur Charpentier 🌻: "Training a single AI model can emit as much carbon as five cars in their lifetimes" https://www.technologyreview.com/s/613630/training-a-single-ai-model-can-emit-as-much-carbon-as-five-cars-in-their-lifetimes/?utm_source=twitter&utm_medium=tr_social&utm_campaign=site_visitor.unpaid.engagement (I knew it was bad.... but that bad !?!) "Deep learning has a terrible carbon footprint" see https://arxiv.org/abs/1906.02243

2 replies, 38 likes


Mark Robinson: Right, I guess we should add (low) carbon footprint as a criteria when benchmarking computational methods then ..

3 replies, 34 likes


Shreya Shankar: https://arxiv.org/pdf/1906.02243.pdf https://t.co/sEyuYr55v2

2 replies, 29 likes


Brandon Rohrer: Small is beautiful. Some machine learning tasks can only feasibly be done by enormous models. But the reflex to improve every model by making it bigger has hidden costs.

1 replies, 27 likes


Pietro Michelucci: Remarkably, the fastest supercomputers can process information about as fast as a human brain. More remarkable, perhaps, is that a supercomputer consumes about 20 million watts, compared to the human brain, which consumes about 20. You have to hand it to nature for efficiency.

1 replies, 26 likes


Jose Javier Garde: Energy and Policy Considerations for Deep Learning in #NLP by Emma Strubell, Ananya Ganesh, Andrew McCallum https://arxiv.org/abs/1906.02243 #deeplearning #ArtificialIntelligence #machinelearning #energy #climatechange #climate #policy #NeuralNetworks #algorithms https://t.co/j3KNLSOr20

0 replies, 25 likes


Emma Strubell: @ACL2019_Italy @ananya__g @andrewmccallum preprint now available! https://arxiv.org/abs/1906.02243

3 replies, 23 likes


Ryan Flynn: I think this gets much less attention than it should

0 replies, 17 likes


Maarten van Smeden: How long before grant agencies will ask you to prognosticate computing costs and carbon footprint for proposals promising "large network AI"? https://arxiv.org/pdf/1906.02243.pdf https://t.co/Kb8L88esKX

2 replies, 16 likes


Alice Coucke: Yes! This should be mandatory. Very interesting talk by @strubell at #acl2019nlp 🌱 👉https://arxiv.org/abs/1906.02243 https://t.co/OYpuluuglu

1 replies, 15 likes


Xander Steenbrugge: "Energy and Policy Considerations for Deep Learning in NLP" They provide some very interesting statistics on the environmental impact of training large Deep Learning models with today's various Cloud Providers! Paper: https://arxiv.org/abs/1906.02243 Article: https://www.technologyreview.com/s/613630/training-a-single-ai-model-can-emit-as-much-carbon-as-five-cars-in-their-lifetimes/ https://t.co/ZsY1ClMa4W

0 replies, 14 likes


Dr Mike Nitabach: But yeah sanctimoniously yelling at people for flying on airplanes is totally awesome. https://twitter.com/EricTopol/status/1216187183402373122

2 replies, 12 likes


Joss Moorkens: @itia_ireland According to this paper the emissions cost for training one system is the equivalent to the lifetime emissions of 6 cars inc fuel. https://arxiv.org/abs/1906.02243

1 replies, 12 likes


Anna Rogers ✈️ #AAAI2020: @strubell at @RealAAAI #AAAI2020 Energy and Policy Considerations for Deep Learning in NLP Paper: https://arxiv.org/pdf/1906.02243.pdf In case anybody needs a reminder, DL research is being environmentally irresponsible, with Google's Meena as the latest offender. @andrewmccallum

1 replies, 10 likes


Denis Newman-Griffis: Fantastic turnout for @strubell's paper on energy consumption in NLP research @ACL2019_Italy (this is only half the room) Paper at https://arxiv.org/abs/1906.02243 #ACL2019 https://t.co/oUNsvl0wio

1 replies, 8 likes


tante: (btw. here are CO2 estimates for training neural nets https://arxiv.org/pdf/1906.02243.pdf )

1 replies, 8 likes


Julie Grollier: "It is estimated that we must cut carbon emissions by half over the next decade, and based on the estimated CO2 emissions listed in Table1, model training and development likely make up a substantial portion of the greenhouse gas emissions" https://arxiv.org/abs/1906.02243 https://t.co/NpNqadgPL6

0 replies, 8 likes


Peter Bloem: While I don't disagree with researching more efficient models, note that this model also _costs_ as much as five cars to train. It's very rare to train such a big model, and it yields an artifact that is reused globally for years.

1 replies, 7 likes


arXiv CS-CL: Energy and Policy Considerations for Deep Learning in NLP http://arxiv.org/abs/1906.02243

0 replies, 7 likes


Steven Lowette: Food for thought. There's no free lunch.

0 replies, 6 likes


Fredros Okumu, PhD: Deep Learning models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware. #deeplearning https://arxiv.org/abs/1906.02243

1 replies, 5 likes


Cait Lamberton: This is interesting. Do we gain enough to make the ecological damage worthwhile? Or is there something missing in this calculation? (I need to read the article to find out what kind of machine-learning model we’re talking about; my bet is it‘s not a logistic regression. :-)

1 replies, 5 likes


Jenny Brennan: Some reading: 🔹Energy and Policy Considerations for Deep Learning in NLP: https://arxiv.org/pdf/1906.02243.pdf 🔹The State of Data Center Energy Use in 2018 @coedethics: https://docs.google.com/document/d/1eCCb3rgqtQxcRwLdTr0P_hCK_drIZrm1Dpb4dlPeG6M/ 🔹Anatomy of an AI system @AINowInstitute: https://anatomyof.ai/ 3/?

1 replies, 5 likes


Anish Mohammed: Energy and Policy Considerations for Deep Learning in NLP < wondering if this was by accident or design, yet another moat for incumbents against challengers #DeepLearning https://arxiv.org/abs/1906.02243

0 replies, 5 likes


Jordi Mas: The ecological cost of training machine learning models (in this case NLP BERT models): "We also see that models emit substantial carbon emissions; training BERT on GPU is roughly equivalent to a trans-American flight." (https://arxiv.org/pdf/1906.02243.pdf)

0 replies, 5 likes


Jumanne Mtambalike: But is it a necessary cost? Or we have alternative.

0 replies, 5 likes


Padraig Cunningham: Here is an estimate of the carbon footprint of Deep Learning; training a DL model could produce as much C02 as 5 cars would in a lifetime - if all the model search / parameter tuning work is taken into account. https://arxiv.org/abs/1906.02243 (Paper presented at ACL2019)

0 replies, 5 likes


Paul Bradshaw: If you're a data journalist exploring #AI/#ML/#NLP, prepare to feel guilty... https://arxiv.org/abs/1906.02243

0 replies, 4 likes


Jean Senellart: @lorenlugosch @SanhEstPasMoi Yes it does. I used the CO₂e calculation model from https://arxiv.org/pdf/1906.02243.pdf. This estimate is based on energy production in USA with 36% non-carbon energy. For China, where the training has maybe (?) been run by #ShannonAI, the figure would be a bit higher.

0 replies, 4 likes


Anna Rogers: The #AcademicTwitter #GreenAI panel from other threads and papers: * @nlpnoah @royschwartz02 @JesseDodge @etzioni https://arxiv.org/abs/1907.10597 * @strubell @andrewmccallum https://arxiv.org/abs/1906.02243 * @alex_lacoste_ @vict0rsch https://arxiv.org/abs/1910.09700 * @bkbrd

1 replies, 3 likes


AI Ethics Lab: Environmental effects of #AI systems should factor in the #ethics analyses. #Carbon footprint of training an AI model: ✈️Roundtrip flight btw NYC & SF: ~2000lbs 🤖 Training transformer w/ NAS: ~626,000lbs! See the paper (2019): https://arxiv.org/pdf/1906.02243.pdf https://www.forbes.com/sites/glenngow/2020/08/21/environmental-sustainability-and-ai/#4b9368157db3

0 replies, 3 likes


Russell Neches: 626,155 pounds of CO2 to train one model? Ouch. Well, I guess now you know why huge tech companies are building custom chips for machine learning. https://arxiv.org/abs/1906.02243

0 replies, 3 likes


arXiv CS-CL: Energy and Policy Considerations for Deep Learning in NLP http://arxiv.org/abs/1906.02243

0 replies, 3 likes


C. Gómez-Rodríguez: @yoavgo @natschluter @afalenska and @strubell et al.'s "Energy and Policy Considerations for Deep Learning in NLP", https://arxiv.org/abs/1906.02243

1 replies, 3 likes


Wolfgang Schröder: #AI #Sustainability #GreenAI #ecology Research published earlier this year found that the training of a neural network creates a carbon dioxide footprint of 284 tonnes - the equivalent of five times the lifespan emissions of a typical car. https://arxiv.org/abs/1906.02243

0 replies, 3 likes


Sanjay Kamath | ಸಂಜಯ್ ಕಾಮತ್: Good paper but not every AI model is a transformer (big) model with neural architecture search. Hope the media doesn't put this out of context.

0 replies, 3 likes


Erik Hamburger: Can #MachineLearning and #Sustainability go together? This study shows how energy intensive all this #ML and #AI is. https://arxiv.org/pdf/1906.02243.pdf

0 replies, 2 likes


Dr William Marshall: Paper: Energy and Policy Considerations for Deep Learning in NLP (Natural Language Processing). Training an #AI model uses a lot of electrical power and leaves a vast carbon footprint. #climatechange https://arxiv.org/pdf/1906.02243.pdf

0 replies, 2 likes


J. Chris Pires: cc #PAGXXVIII #PAG2020 #AI Machine Learning

0 replies, 2 likes


Manyvip: A Deep Learning Process Can Emit 284.000 kilograms of Carbon Dioxide (CO2). Download PDF. https://arxiv.org/abs/1906.02243 https://t.co/OGOT6UXFB4

0 replies, 2 likes


Dave Costenaro: Interesting paper: "Energy and Policy Considerations for Deep Learning in NLP." Training 1 big NN model has the carbon footprint of 5 cars over their lifetimes. (https://arxiv.org/abs/1906.02243). Compute is cheap...but not free, so please give efficient code some thought!

0 replies, 2 likes


Beril Sirmacek 🦋: From now on, the #artificialintelligence frameworks will not be judged by their #performance but by their #energy labels. #carbonfootprint #climatechange #co2 https://arxiv.org/abs/1906.02243 https://t.co/NYN5xqJnMq

1 replies, 2 likes


Patrick Burr: #AI comes at an environmental cost. The largest bearer of it's externalities is our planet. I can't vouch for the accuracy of these findings, but I have similar trends quite few times now. Running loads of CPUs directly increases CO2 emissions and resources consumption.

1 replies, 2 likes


Tim Heiler: The average deep learning model using fossil fuel releases around 78,000 pounds of carbon. That’s more than half of a car's output from assembly line to landfill. According to this paper: https://arxiv.org/abs/1906.02243 #AI #machinelearning #ClimateChange #ClimateEmergency https://t.co/27GQVlH1wK

0 replies, 1 likes


Philippe Durance: Recent progress in training #neuralnetworks depends on the availability of exceptionally large computational resources that necessitate similarly substantial #energy consumption https://arxiv.org/abs/1906.02243?utm_campaign=the_algorithm.unpaid.engagement&utm_source=hs_email&utm_medium=email&utm_content=73464008&_hsenc=p2ANqtz-_Fe6QnHVjx60Rmn8sSlVsG30Q0TJFvIXv2ykzz8aVKdt_RV6sUronq35AtaM5iZ1YOF8qIQdICVUflzM_vHIREtVblgQ&_hsmi=73464008 @Cornell

0 replies, 1 likes


Libby Hemphill, PhD: @danieljkelley Just starting the read, but the environmental impact of our models is definitely something we talk about on my team. @strubell had a great paper at #ACL2019 about neural models and their costs: https://arxiv.org/abs/1906.02243

0 replies, 1 likes


Richard Rathe, MD: Reminds me of the energy needed to support #cyptocurrencies and so-called "mining". Enough electricity to keep whole cities going for months/years!!

0 replies, 1 likes


Swadhin | স্বাধীন: @krismicinski A recent related paper focusing on NLP model and energy : https://arxiv.org/abs/1906.02243 . Emma discusses about this in detail in TWIMLAI podcast this week.

0 replies, 1 likes


Andrés Murcia: La huella de carbono del Deep Learning - "Runs on energy-intensive computer chips, can emit more than 626,000 pounds of carbon dioxide equivalent, nearly five times the lifetime emissions of the average American car." - https://arxiv.org/abs/1906.02243?utm_campaign=the_algorithm.unpaid.engagement&utm_source=hs_email&utm_medium=email&utm_content=73464008&_hsenc=p2ANqtz-_46saoiHzXONwwvcO8_1mRilORNzze1VMZK13OjGfGio6b6T1fa4hK60qYibywgomX5-w8tRl0vrOP0HnIcSWyXcS9wQ&_hsmi=73464009 https://t.co/mR3lVt6tNp

1 replies, 1 likes


Alasdair Allan: @swardley I do have some issues with the broader applicability of their analysis, but here's the link to the Strubell, Ganesh & McCallum (2019) paper with the life-cycle analysis of #MachineLearning training I talked about at the start of my #OSCON talk, https://arxiv.org/abs/1906.02243.

0 replies, 1 likes


Juan A. Botía: Next time you have some desire to do a blind search for your deep ann model best parameters think twice!

0 replies, 1 likes


Eduardo Eyras: Given the formula CO2e = 0.954pt, pretty much all current data science will end up burning the planet https://arxiv.org/abs/1906.02243

0 replies, 1 likes


Jordan Foley: Really interesting work on the potential environmental implications of machine learning and AI. Ive seen lots of important convos about the ethical dimensions of these technologies but few that center questions like these. https://arxiv.org/abs/1906.02243

0 replies, 1 likes


Ken Figueredo: #Green credentials and #MachineLearning https://arxiv.org/pdf/1906.02243.pdf https://t.co/3zJ3FxxZxi

0 replies, 1 likes


World Ethical Data Forum: Here's the UMass paper, if the numbers interest you: https://arxiv.org/pdf/1906.02243.pdf Thanks to @InfoMgmtExec for pointing out the original post wasn't clear enough.

0 replies, 1 likes


Nate Jue: Colleague sent this article to me and it's making me think more and more about the environmental impacts of my computational work and associated ethical decisions. How many of us computational biologists even have this on our radar? I sure haven't.😑 https://arxiv.org/abs/1906.02243

0 replies, 1 likes


Miguel Luengo-Oroz: Some of the refs I shared: review on AI applications for climate https://arxiv.org/pdf/1906.05433.pdf ; how AI models grow exponentially https://openai.com/blog/ai-and-compute/ ; CO2 footprint of an AI system https://arxiv.org/pdf/1906.02243.pdf ; sustainability as a principle of AI development https://rdcu.be/bUYS1

0 replies, 1 likes


Peter Coffee: "Large networks trained on abundant data depend on the availability of exceptionally large computational & energy resources. We propose actionable recommendations to reduce costs and improve equity in NLP research and practice." [PDF] https://arxiv.org/pdf/1906.02243.pdf

0 replies, 1 likes


GetzlerChem: @Chemjobber See also the surprising and staggering cost of deep learning. (preprint, so caveat emptor, etc) https://arxiv.org/abs/1906.02243 https://t.co/ZxC7snogK9

0 replies, 1 likes


Harriet Kingaby: (9/16) And then there's #climatechange. A study from the University of Massachusetts reported that training one AI model produced 300,000 kilograms of CO2. That's roughly the equivalent of 125 round trip flights from New York to Beijing https://arxiv.org/abs/1906.02243. https://t.co/vViqmt5Ck4

1 replies, 1 likes


DOSE Engineering: Data centers and cloud computing providers need to up their use of renewable energy in order to meet the high energy demands of CPU/GPU/TPU by artificial intelligence/deep learning. #greenenergy #cloudcomputing #datacenters #AI https://arxiv.org/pdf/1906.02243.pdf https://t.co/8SssmMf7Hl

0 replies, 1 likes


Charles Starrett: We know computation like this, not to mention #blockchain (*shudder*), is worsening our #climate crisis. Why aren't we pushing harder for data centers to have their own solar/wind farms? — Energy and Policy Considerations for Deep Learning in NLP https://arxiv.org/abs/1906.02243

0 replies, 1 likes


Nathaniel Bullard: First: an AI model doesn't run on coal or gas; it runs on electricity, and its carbon will be a direct result of the power mix used to energize the data centers it runs on. The paper https://arxiv.org/pdf/1906.02243.pdf gets that right...

1 replies, 0 likes


Lancelot PECQUET: #environment - Training a single #AI model can emit as much #carbon as five cars in their lifetimes https://arxiv.org/pdf/1906.02243.pdf https://t.co/64HGDETeML

1 replies, 0 likes


Zachary Lipton: @VishnuBoddeti So far, I have yet to be convinced of neural architecture search as a research direction but my degree of certainty is not high. To date, NAS requires 1000s× more resources w/o qualitatively stronger results. See paper by @strubell on environmental impact—https://arxiv.org/abs/1906.02243

1 replies, 0 likes


Emily Hopkins: @emiliagogu @IEEEorg 5. Diversity, non-discrimination, fairness accessibility, bias, competing interests & objectives #ismir2019 6. Societal and environmental well-being sustainability and benefit to future generations energy use of deep learning: https://arxiv.org/abs/1906.02243

1 replies, 0 likes


Cory Doctorow #BLM: #1yrago Training a modest machine-learning model uses more carbon than the manufacturing and lifetime use of five automobiles https://arxiv.org/abs/1906.02243 11/ https://t.co/YoK8LYVY1a

1 replies, 0 likes


Content

Found on Jan 12 2020 at https://arxiv.org/pdf/1906.02243.pdf

PDF content of a computer science paper: Energy and Policy Considerations for Deep Learning in NLP