Query Response

Query data

Score

Prompt

System Prompt

You are a reporter for a major world newspaper. Write your response as if you were writing a short, high-quality news article for your paper. Limit your response to one paragraph. Use the following article for context: LLaVA-CoT Shows How to Achieve Structured, Autonomous Reasoning in Vision Language Models - InfoQ<link rel="stylesheet" type="text/css" href="https://cdn.infoq.com/statics_s1_20241119073511/styles/style_en.css"/> <link rel="stylesheet" href="https://cdn.infoq.com/statics_s1_20241119073511/styles/icons.css"> <link rel="stylesheet" type="text/css" media="screen" href="https://cdn.infoq.com/statics_s1_20241119073511/styles/style_extra.css"/><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-W9GJ5DL" height="0" width="0" style="display:none;visibility:hidden"></iframe>BTInfoQ Software Architects' NewsletterA monthly overview of things you need to know as an architect or aspiring architect.View an exampleEnter your e-mail addressSelect your countrySelect a countryI consent to InfoQ.com handling my data as explained in thisPrivacy Notice.We protect your privacy.CloseToggle NavigationFacilitating the Spread of Knowledge and Innovation in Professional Software DevelopmentEnglish editionEnglish editionChinese editionJapanese editionFrench editionWrite for InfoQSearchSign Up / LoginEmailPasswordForgot password ?InfoQ Account EmailBack to loginResend ActivationBack to loginLogin with:GoogleMicrosoftTwitterFacebookDon't have an InfoQ account?Sign UpLogo - Back to homepageNewsArticlesPresentationsPodcastsGuidesTopicsDevelopmentJavaKotlin.NetC#SwiftGoRustJavaScriptFeatured in DevelopmentBeyond the Breach: Proactive Defense in the Age of Advanced ThreatsMichael Brunton-Spall discusses some of the most advanced attacks that are in the public domain, mostly attributed in public by commercial organizations.All in developmentArchitecture & DesignArchitectureEnterprise ArchitectureScalability/PerformanceDesignCase StudiesMicroservicesService MeshPatternsSecurityFeatured in Architecture & DesignTransforming Legacy Healthcare Systems: A Journey to Cloud-Native ArchitectureDiscover how Livi navigated the complexities of transitioning MJog, a legacy healthcare system, to a cloud-native architecture, sharing valuable insights for successful tech modernization. Our experience illustrates that transitioning from legacy systems to cloud-based microservices is not a one-time project but an ongoing journey.All in architecture-designAI, ML & Data EngineeringBig DataMachine LearningNoSQLDatabaseData AnalyticsStreamingFeatured in AI, ML & Data EngineeringNavigating LLM Deployment: Tips, Tricks, and TechniquesMeryem Arik discusses some of the best practices in model optimization, serving and monitoring - with practical tips and real case-studies.All in ai-ml-data-engCulture & MethodsAgileDiversityLeadershipLean/KanbanPersonal GrowthScrumSociocracySoftware CraftmanshipTeam CollaborationTestingUXFeatured in Culture & MethodsParticipatory Leadership and Developing a Culture of Psychological SafetyIn this podcast Shane Hastie, Lead Editor for Culture & Methods spoke to Nick Takavadii about participatory leadership practices and how to cultivate a workplace environment with psychological safety.All in culture-methodsDevOpsInfrastructureContinuous DeliveryAutomationContainersCloudObservabilityFeatured in DevOpsFrom Local to Production: A Modern Developer’s Journey Towards KubernetesUrvashi Mohnani discusses the full developer experience of writing an application, containerizing it locally, deploying it to a Kubernetes cluster, and debugging Kubernetes applications locally.All in devopsEventsHelpful linksAbout InfoQInfoQ EditorsWrite for InfoQAbout C4MediaDiversityChoose your languageEn中文日本FrQCon San FranciscoLevel up your software skills by uncovering the emerging trends you should focus on. Register now.QCon LondonDiscover emerging trends, insights, and real-world best practices in software development & tech leadership. Join now.InfoQ Dev Summit BostonLearn how senior software developers are solving the challenges you face. Register now with early bird tickets.The Software Architects' NewsletterYour monthly guide to all the topics, technologies and techniques that every professional needs to know about. Subscribe for free.InfoQ HomepageNewsLLaVA-CoT Shows How to Achieve Structured, Autonomous Reasoning in Vision Language ModelsAI, ML & Data EngineeringLLaVA-CoT Shows How to Achieve Structured, Autonomous Reasoning in Vision Language ModelsLikeBookmarksNov 24, 20242 min readbySergio De SimoneWrite for InfoQFeed your curiosity.Help 550k+ globalsenior developerseach month stay ahead.Get in touchResearchers from several Chinese institutionsfine-tuned Llama-3.2-11B-Vision-Instruct to improve its ability to solve multimodal reasoning problemsby going beyond the direct-response or chain-of-thought (coT) approaches to reason step by step in a structured way. NamedLLava-CoT, the new model outperforms its base model and proves better than larger models, including Gemini-1.5-pro, GPT-4o-mini, and Llama-3.2-90B-Vision-Instruct, on a number of benchmarks.According to the Chinese researchers, one reason why visual language models (VLMs) often hallucinate or produce error is the lack of systematic and structured reasoning:Specifically, by referringsystematic, we mean that the model does not generate a direct reasoning chain but instead engages in multistage reasoning.Structured, on the other hand, refers to the model’s ability to clearly identify the reasoning stage it is in and understand the primary task to be addressed at each stage.The approach taken by the research authors consists in designing LLaVA-CoT so it reasons through four stages: asummary, where the model summarized the current task; acaption, which is a description of the relevant parts of an image;reasoning, where the model analyzes the question; andconclusion, which provides a final response based on the reasoning stage. In other words, the model first organizes the problem and all known information, then it carries through a detailed thought process, and finally derives a conclusion.To make this possible, the researchers constructed a specific dataset, LLaVA-o1-100k, by using GPT-4o to generate responses stage by stage. The custom dataset includes data from both general-purpose visual question answer (VQA) datasets as well as science-targeted VQA datasets. They used then the generated dataset to perform a full parameter fine-tuning of Llama-3.2-11B-Vision-Instruct in a supervised approach.Additionally, LLaVA-CoT uses a novel approach to efficient inference time scaling. Instead of usingbeam searchat the sentence level, they use it at the stage level to generate multiple candidate results at each stage. The best potential result is then selected to continue the generation process at the next stage. According to the authors, using inference time scaling makes it possible for the model to arrive at a concrete answer during the reasoning process and retain it for the final stage. Lacking this, the model could need to make a guess for the final stage, possibly leading to incorrect results.Stage-level beam search, which is made possible by the structured output design of [LLaVA-CoT], is an effective and powerful approach for inference time scaling.To assess their approach, the researchers compared LLaVA-CoT performance to both its base model and other models. They found LLaVA-CoT provides notable improvements across general VQA, mathematical reasoning, scientific VQA, and hallucination control tasks in comparison to its base model. Additionally, LLaVA-CoT appears to outperform many open-source models of similar or even larger sizes, such as InternVL2-8B, Ovis1.5-Gemma2-9B, MiniCPM-V2.6-8B, Llama-3.2-90B-Vision-Instruct, and VILA-1.5-40B, as well as closed-source models such as GPT-4o-mini and Gemini-1.5-pro.LLaVA-Cot is available on Hugging Face, while the LLaVA-o1-100k dataset will be made public in future, say the authors.A Web app is also availablewhich allows to upload an image and start chatting about it.About the AuthorSergio De SimoneShow moreShow lessRate this ArticleAdoptionStyleAuthor ContactedThis content is in theAI, ML & Data EngineeringtopicRelated Topics:DevelopmentAI, ML & Data EngineeringComputer VisionLarge language modelsRelated EditorialRelated Sponsored ContentPopular across InfoQAWS Amplify and Amazon S3 Integration Simplifies Static Website HostingAurora Limitless: AWS Introduces New PostgreSQL Database with Automated Horizontal ScalingMeta Releases NotebookLlama: Open-Source PDF to Podcast ToolkitHow Recall.ai Saved $1M on AWS by Eliminating WebSocketsSpring Framework 6.2 and Spring Boot 3.4 Improve Containers, Actuators ahead of New 2025 GenerationsTrends in Engineering Leadership: Observability, Agile Backlash, and Building Autonomous Teams<div class="widget related__content article__widget"> <h3 class="widget__heading">Related Content</h3> <ul class="no-style cards" data-horizontal="true" data-size="xs" data-tax=""> </ul> </div>The InfoQNewsletterA round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers.View an exampleEnter your e-mail addressSelect your countrySelect a countryI consent to InfoQ.com handling my data as explained in thisPrivacy Notice.We protect your privacy.DevelopmentCarle Lerche Talking at QCon SF about Rust: a Productive Language for Writing Database ApplicationsGoogle Introduces Gemini AI Features to Android StudioGitHub Universe 2024 Unveils AI Innovations and Developer-Centric ToolsArchitecture & DesignNetflix Rolls Out Service-Level Prioritized Load Shedding to Improve ResiliencyTransforming Legacy Healthcare Systems: A Journey to Cloud-Native ArchitectureNew "Laws" Announced at iSAQB Software Architecture GatheringCulture & MethodsParticipatory Leadership and Developing a Culture of Psychological SafetyHow to Delight Your Developers with User-Centric Platforms and PracticesTrends in Engineering Leadership: Observability, Agile Backlash, and Building Autonomous TeamsAI, ML & Data EngineeringLLaVA-CoT Shows How to Achieve Structured, Autonomous Reasoning in Vision Language ModelsMicrosoft Announces General Availability of Fabric API for GraphQLVercel Expands AI Toolkit with AI SDK 4.0 UpdateDevOpsFrom Local to Production: A Modern Developer’s Journey Towards KubernetesTimescale Bolsters AI-Ready PostgreSQL with pgai VectorizerKubeCon + CloudNativeCon NA 2024: Key Announcements and Projects UpdatesThe InfoQNewsletterA round-up of last week’s content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers.View an exampleGet a quick overview of content published on a variety of innovator and early adopter technologiesLearn what you don’t know that you don’t knowStay up to date with the latest information from the topics you are interested inEnter your e-mail addressSelect your countrySelect a countryI consent to InfoQ.com handling my data as explained in thisPrivacy Notice.We protect your privacy.QCon San FranciscoNovember 18-22, 2024.QCon San Francisco International Software Conference returns on November 18-24. More than 1000 software professionals will join together and learn about the emerging trends they should pay attention to in 2024, how to adopt them, how to avoid pitfalls, and how to embrace the best practices.Join the experience and get implementable ideas to shape your projects that last beyond the conference.Register NowHomeCreate accountLoginQCon ConferencesEventsWrite for InfoQInfoQ EditorsAbout InfoQAbout C4MediaMedia KitInfoQ Developer Marketing BlogDiversityEventsQCon San FranciscoNOVEMBER 18-22, 2024QCon LondonAPRIL 7-9, 2025InfoQ Dev Summit BostonJune 9-10, 2025Follow us onYoutube223K FollowersLinkedin21K FollowersRSS19K ReadersX53.4k FollowersFacebook21K LikesAlexaNewStay in the knowThe InfoQ PodcastEngineering Culture PodcastThe Software Architects' NewsletterGeneral Feedbackfeedback@infoq.comAdvertisingsales@infoq.comEditorialeditors@infoq.comMarketingmarketing@infoq.comInfoQ.com and all content copyright © 2006-2024 C4Media Inc.Privacy Notice,Terms And Conditions,Cookie PolicyBT<img height="1" width="1" src="https://www.facebook.com/tr?id=842388869148196&ev=PageView&noscript=1"/>

User Query

deep learninf

Result

Source

https://www.infoq.com/news/2024/11/llava-cot-visual-language-model/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=Development

Response

In a significant breakthrough in artificial intelligence, researchers from various Chinese institutions have developed LLaVA-CoT, a vision-language model that enhances structured, autonomous reasoning in AI systems. By refining the Llama-3.2-11B-Vision-Instruct model, LLaVA-CoT outperforms its predecessors and several larger models in tackling multimodal reasoning tasks. This new approach involves a four-stage reasoning process: summarizing the task, captioning relevant image parts, analyzing the question, and drawing conclusions. By shifting from a direct response method to multistage reasoning, LLaVA-CoT addresses common issues such as hallucinations and errors in visual language models. The innovation further uses a unique beam search strategy at the stage level for efficient inference time scaling, allowing the model to maintain accuracy in generating responses. Public availability of LLaVA-CoT on platforms like Hugging Face, along with its dataset shortly, marks a leap forward in the AI domain, offering researchers a robust tool for improving visual question-answering and reasoning capabilities.