RNN’s are Schmidhuber’s Revenge

Can't read or see images? View this email in a browser

When Jürgen Schmidhuber said that Recurrent Neural Networks (RNNs) are all we need, not many believed him. But it seems like researchers are realising that it was true all along. Moreover, apart from matrix multiplication, some researchers suggest using addition to increase the efficiency of neural networks. Your AI Memelord, Mohit Pandey, is here to explain this in the easiest way possible. Were RNNs All We Needed? The researchers asked this question to revive traditional RNNs, including LSTMs and Gated Recurrent Units (GRUs). They concluded that by removing the hidden state dependencies from their input, forgetting them, and updating gates, LSTMs and GRUs no longer need BPTT and can be efficiently trained in parallel.  The minimal versions of LSTMs and GRUs called minLSTM and minGRU, are stripped-down versions of the original models. Unlike traditional RNNs, they can be trained simultaneously using the parallel scan algorithm, significantly speeding up training time.  How fast are they? These two models use significantly fewer parameters than their traditional counterparts, resulting in minGRU and minLSTM being 175x and 235x faster per training step than traditional GRUs and LSTMs for a context length of 512 tokens. With Transformers, you can also fetch any previous information at any time which is quite helpful. Meanwhile, RNNs are constantly updating and overwriting their memory, because they are without BPTT. It means they need to be able to predict what will be useful in order to store it for later.  This is a massive advantage for Transformers in interactive use cases like in ChatGPT when the users give context and ask questions in multiple turns.  Why Does Everyone Want Transformers to Fail? This is not a new phenomenon. In fact, researchers predicted the same in 2019 in the research paper titled ‘Single-Headed Attention RNN: Stop Thinking With Your Head’.   It demonstrated near SOTA results using LSTMs, a modified version of RNNs, which makes it easier to remember past data in memory, suggesting that the “Transformer hype” may be overblown.  Transformers require the entire sequence to be stored in memory, but that won’t be realistic when the models become highly multimodal and start taking in a lot of information, such as images, video, sound, etc,  at once.   RNNs don’t have this limitation by design, they only need the data corresponding to the current timestep. Hence, they will end up scaling better in the context of multimodality, even if Transformers stays better in terms of accuracy.  The hybrid approach of combining the strengths of RNNs with lessons learned from the Transformer era could point the way forward for sequence modelling tasks. The ability to train these models efficiently on large datasets while maintaining the conceptual simplicity of RNNs is particularly appealing.  Read the full story here. Meanwhile, Can We Reduce the Power Consumption of Neural Networks? The power consumption of such AI models has been discussed intensively over the last few years. And as we all know, it’s huge!  But there is a way to scale it down. A research paper by BitEnergy AI, titled, ‘Addition is All You Need: For Energy Efficient Language Models’ mentioned that multiplying floating point numbers consumes significantly more energy than integer operations.   The paper states that multiplying two 32-bit floating point numbers (fp32) costs four times more energy than adding two fp32 numbers and 37 times more than adding two 32-bit integers.  Neural networks typically perform computations using standard floating-point multiplication, which is computationally expensive and energy-intensive, especially for LLMs, which typically run over billions of parameters. These operations consume significant computational resources and energy, particularly in attention mechanisms and matrix multiplications.   The researchers have proposed a new technique called linear-complexity multiplication (L-Mul), which solves the problem of energy-intensive floating point multiplications in large neural networks.   It uses straightforward bit operations and additions to avoid complicated multiplication of parts of the numbers (mantissa) and tricky rounding steps.  This approach not only reduces the computational cost, but also potentially decreases energy consumption by up to 95%,  for element-wise floating-point tensor multiplications and 80% for dot products while maintaining comparable or even superior precision to 8-bit floating-point operations in many cases.  Click here to read the full story. AI Bytes Google Cloud announced that it is partnering with Sequoia Capital, which will allow its portfolio companies to access Google’s various cloud services, credits, and enhanced support. AI startup Sierra reportedly seeks over $4 billion in funding, which would quadruple the company’s valuation compared to its $1 billion deal earlier this year. NVIDIA and Foxconn have unveiled plans to construct Taiwan’s largest supercomputer, housed at the Hon Hai Kaohsiung Supercomputing Center.  David Baker, an American biochemist and computational biologist, and Demis Hassabis and John M Jumper, two Google DeepMind scientists, have been awarded the Nobel Prize in Chemistry 2024 by The Royal Swedish Academy of Sciences.  At the Go Get Zero conference in London, Uber announced the upcoming launch of an AI assistant powered by OpenAI’s GPT-4. Atlassian, an American-Australian software company, announced that its recently introduced team-building AI, Rovo, will now be available for all users.  OpenAI announced that its team received the first engineering builds of NVIDIA’s DGX B200. These new builds promise three times faster training speeds and fifteen times greater inference performance than previous models. Fraud and Risk Detection in the LLM Era LLMs are powerful tools for detecting financial fraud—but they’re also creating new risks. Are your defences ready? Discover how to stay ahead with proactive strategies!  Register here. NVIDIA AI Summit India Join the NVIDIA AI Summit India from October 23–25, 2024, at the Jio World Convention Centre in Mumbai to explore AI innovations across generative AI, robotics, supercomputing, and more, with 70% of use cases addressing India's grand challenges.  Don't miss the Fireside Chat with NVIDIA CEO Jensen Huang on October 24.  Register here to be part of this transformative event. AI Conclave Wonders AIM is bringing Cypher 2024 to Silicon Valley! And we are all geared up, having just wrapped up the biggest AI conference in Bengaluru last month.  Where: Santa Clara, California. When: November 21-22, 2024  Register now!  Enjoying Sector 6 (formerly AIM Daily XO)? Share it with colleagues or friends – they can sign up here.   We love hearing from our readers! Have thoughts on our new format? Questions, comments, or ideas are always welcome. If there’s a specific topic in AI or analytics that you're curious about, tell us!  Reach out to us at info@analyticsindiamag.com. Stay tuned for more insights in our next edition! Curated with ♥️ in Namma Bengaluru

                                                           

This email was sent by info@aimmediahouse.com to alexvarboffin.abbb@blogger.com Not interested? Unsubscribe | Manage Preference | Update profile Analytics India Magazine | 280, 2nd floor, 5th Main, 15 A cross, Sector 6, HSR layout Bengaluru, Karnataka 560102