| | | | | | | | | | | | | | To reach this conclusion, over 100 NLP researchers were asked to come up with new ideas and review both LLM- and human-generated ideas without knowing their source. The results showed that LLM ideas were considered more innovative (with statistical significance, p < 0.05), although they were rated slightly lower in terms of feasibility.
The approach is similar to that of the Japanese AI startup, Sakana AI’s AI Scientist, which automates the entire research lifecycle. It generates novel research ideas, writes necessary code, executes experiments, summarises results, visualises data, and presents its findings in a complete scientific manuscript.
Interestingly, the startup claimed that each idea is implemented and developed into a full paper at approximately $15 per paper.
Generating new ideas is relatively easy for LLMs, thanks to their extensive training on large datasets and ability to combine various concepts. However, they continue to face challenges with advanced reasoning.
Meanwhile, OpenAI is preparing to release its new model, Strawberry, which is expected to offer improved reasoning capabilities.
Chai Discovery, a biology startup founded by a former OpenAI employee, recently introduced Chai-1, an advanced foundation model that predicts molecular structures crucial for drug discovery. Innovations like these show that LLMs are close to driving significant research breakthroughs.
“The ability of LLMs to combine concepts from vast datasets in ways not typically thought of by humans can lead to ideas that are considered more novel. This might be because LLMs aren’t constrained by the same cognitive biases or conventional thinking patterns that humans have,” said DigitalVibes.ai founder Anthony Scaffeo.
He added that LLMs can make connections across different fields or unrelated data points, which might not be intuitive or immediately obvious to human experts.
Another startup, EvolutionaryScale, backed by Amazon and NVIDIA, is using LLM-based models like ESM3 to develop novel proteins for scientific research, aiming to revolutionise drug discovery and materials science through AI-driven protein engineering. | | | | | | | “My student’s comment on the paper about LLMs generating more novel research ideas than humans is making the rounds. I think this says more about NLP researchers than about LLMs. Ouch,” joked Subbarao Kambhampati, professor of computer science at Arizona State University.
“I am not gonna let no LLM beat me in generating novel NLP research ideas,” he quipped. Interestingly, Kambhampati has been quite vocal about LLMs being bad at reasoning and planning.
He said that models like GPTs 3, 3.5, and 4 are poor at planning and reasoning, which he believes involves time and action. According to him, these models struggle with transitive and deductive closure, with the latter involving the more complex task of deducing new facts from the existing ones.
Today researchers have not experimented much with LLMs to generate novel ideas, instead they have been predominantly using it to review research papers.
Apparently, Meta AI chief Yann LeCun argues that while LLMs cannot reason and plan, they are still a good tool for reviewing papers. “[Human] reviewers should be able to use the tools they want to help them write reviews. The quality of their reviews should be assessed based on the result, not the process,” he said.
Meta AI launched Galactica, an LLM for research, in November 2022, just weeks before ChatGPT. However, it was taken down after three days due to criticism over generating misleading or offensive information. LeCun remains unhappy about it to this day. | | | | | | | However, not everyone agrees with LeCun. | | | | | “AI-generated reviews of scientific papers are increasing, vacuous, and need to be stopped quickly. They reduce the author’s trust in the review process. Proposal: someone who is judged to have submitted such a review is banned from submitting to the same conference/journal for two years,” said Micheal Black, director, Max Planck Institute for Intelligent Systems.
Adding to this perspective, Mukur Gupta, an applied scientist at Apple, recounts his frustrating experience with an LLM-generated review.
“I love AI as an assistant. But after getting an LLM-generated review for my NeurIPS paper last month (which was total crap and useless), I’m a little sceptical about AI discovering true novelty,” said Gupta.
He explained that LLMs could be a game-changer for interdisciplinary research or for uncovering new problems in fields where human experts, limited by their working memory and attention span, may struggle to grasp more than a handful of domains.
“LLMs, with their ever-expanding knowledge base, offer the potential for cross-pollination of ideas. But when it comes to deep, niche, and fundamental breakthroughs, I’m not buying it—hence my disappointment with that NeurIPS review,” he added.
Lately, there has been a growing trend of researchers using LLMs to write papers. According to recent data, the use of the term ‘delve’ in the abstracts gradually increased through 2022, jumped noticeably in 2023 (when ChatGPT became widely available), and has continued to rise in 2024.
The future of research should be a collaboration between humans and LLMs to generate truly innovative ideas. According to Stanford’s paper, human ideas often prioritise feasibility and effectiveness over novelty and excitement, which can limit their creativity.
On the other hand, LLMs struggle to judge the quality of ideas. By combining the strengths of both humans and LLMs, we can pave the way for exciting research.
Enjoy the full story here. | | | | | Why Cursor is Ahead of the Curve | | | | | | | | | | | Oracle recently unveiled the world’s first zettascale AI supercomputer, powered by 131,072 NVIDIA Blackwell GPUs, delivering 2.4 zettaFLOPS of performance and positioning itself as the leader in scalable AI infrastructure for advanced AI workloads, surpassing competitors like AWS, Microsoft Azure, and Google Cloud. Oracle has also unveiled over 50 AI agents integrated into its Fusion Cloud Applications Suite, automating tasks across finance, HR, supply chain, and more. Hume AI introduced EVI 2, a voice-to-voice AI model designed for human-like, emotionally intelligent conversations with multilingual support, adaptive personalities, and a focus on preserving voice identity without cloning. Yotta Data Services has launched Shambho, an AI startup accelerator in partnership with Nasscom and Telangana AI Mission, offering cloud credits, mentorship, and technical support. Klarna CEO Sebastian Siemiatkowski recently announced the company’s decision to end relationships with Salesforce and Workday, streamlining its tech stack through AI initiatives to enhance efficiency and reduce the workforce by 50%. Accel has launched Accel Atoms 4.0, focusing on AI and Bharat cohorts to support pre-seed startups, offering up to $1 million in investment, mentorship, and resources for AI innovations and scalable solutions tailored to Tier 2 and 3 cities in India. Progress has acquired ShareFile, a cloud-based collaboration platform, for $875 million, adding over $240 million in annual revenue and 86,000 customers to its portfolio. | | | | | | | | | Join the NVIDIA AI Summit India from October 23–25, 2024, at the Jio World Convention Centre in Mumbai to explore AI innovations across generative AI, robotics, supercomputing, and more, with 70% of use cases addressing India's grand challenges. Don't miss the Fireside Chat with NVIDIA CEO Jensen Huang on October 24. | | | | | AIM & NVIDIA Present DevPalooza 4.0: The Ultimate Developer Meetup in Bengaluru | | | | | | | Join us at DevPalooza 4.0 in Bengaluru, powered by AIM and NVIDIA, to dive into hands-on generative AI workshops, explore applications, and network with AI professionals—click here to register and secure your spot! | | | | | | | | | Cypher 2024 marks a significant expansion as it celebrates its 8th edition by branching out to the USA in addition to its already established presence in India. Browse through the links below to learn more about the different editions of Cypher 2024. These links will guide you to comprehensive event information, including agendas, speakers, registration details, and more. | | | | | Enjoying Sector 6 (formerly AIM Daily XO)? Share it with colleagues or friends – they can sign up here. We love hearing from our readers! Have thoughts on our new format? Questions, comments, or ideas are always welcome. If there’s a specific topic in AI or analytics that you're curious about, tell us! Reach out to us at info@analyticsindiamag.com. Stay tuned for more insights in our next edition!
Curated with ♥️ in Namma Bengaluru | | | | | | | | |
Комментариев нет:
Отправить комментарий
Примечание. Отправлять комментарии могут только участники этого блога.