LlamaIndex Update — 07/11/2023

Greetings once again, LlamaIndex community!

Welcome back to our second installment in the LlamaIndex Update series. In our ongoing commitment to keep you informed and engaged with our rapidly evolving open-source project, this blog post brings you more exciting updates on features, webinars, hackathons, and community events.

Building on the foundation of our inaugural post, we will continue to strive to keep both our long-standing contributors and fresh faces synced with our progress. We aim to not just inform but also inspire you to partake in our collective journey towards growth and innovation.

Without further delay, let’s delve into the latest happenings in this edition of the LlamaIndex Update.

Features And Integrations:

1. LlamaIndex’s partnership with Anyscale uses the Ray platform to boost performance and deployment. It accelerates LlamaIndex’s operations by a factor of ten and streamlines deployment to production servers. The core Ray Distributed Toolkit aids in efficient task parallelization, while Ray Serve ensures easy deployment of query engines to production. Blogpost, Tweet
2. LlamaIndex enhanced metadata representation in documents. The extra_info and node_info fields are now replaced with a metadata dictionary. This facilitates precise control over data and allows users to exclude metadata keys during embedding or LLM prediction. This boosts LLM and retrieval performance and offers customizable metadata injection, formatting, and template creation. Docs, Tweet
3. LlamaIndex supports both Text Completion API, involving output parsing and input prompt modification, and Structured API, requiring input function signatures and output conversion. Despite Structured API being easier to use, its limited availability keeps Text Completion API relevant. Both are supported by LlamaIndex’s PydanticProgram. Docs, Tweet
4. LlamaIndex now collaborates with Chainlit.io, facilitating swift construction of advanced chat UIs for any LLM app. This integration, beyond providing a basic chat interface, also logs intermediate results and sources. Blogpost, Tweet
5. LlamaIndex now incorporates the DePlot model for interpreting charts and plots in QA/chatbot applications. Primarily effective for simple charts, such as bar charts and time series, DePlot converts these visuals into text format for easy embedding, indexing, and usage in downstream applications. This functionality is now accessible via the LlamaHub data loader, expanding LlamaIndex’s capabilities for diverse applications. Docs, Tweet
6. LlamaIndex now incorporates the Github Issues reader, which allows for comprehensive loading and querying of issues from any GitHub repository. Additionally, the Sitemap Loader reader enables users to read all webpages from a specified sitemap. Tweet
7. LlamaIndex introduces the ContextRetrieverOpenAIAgent feature, which enhances tool picking by incorporating more context from user messages. It performs a retrieval step before the LLM call, ensuring increased reliability and better mapping of queries to the right tools, especially in the presence of domain-specific terms. Unlike a “retrieval tool”, this feature guarantees retrieval before any action is taken. Docs, Tweet
8. LlamaIndex now features code-based extraction for efficient data extraction from arbitrary text. This feature includes a “Fit” step to generate functions based on training data, and an “Inference” step to run these functions on new data. It offers two versions: DFEvaporateProgram for extracting one value per field from a text, and MultiValueEvaporateProgram for extracting multiple values per field. This feature can be used to extract structured data from raw HTML sources and also offers the ability to identify salient fields in a text given a topic. Docs, Tweet
9. LlamaIndex has significantly improved its text-to-SQL capabilities, offering a “Default” SQL query engine and an SQL query engine with an object index for handling large table schemas. These upgrades simplify the process, requiring only a SQL database for the default engine and enabling indexing of large table schemas with the ObjectIndex. Additionally, LlamaIndex now also integrates with duckdb, further enhancing the SQL querying process. Docs_SQL, Docs_duckdb, Tweet
10. LlamaIndex 0.7.0 enhances modularity for LLM app development. It includes native LLM abstractions for platforms like OpenAI and Hugging Face, a standalone Response Synthesis module, and improved Document Metadata Management. These abstractions can be used independently or integrated into indices/query engines. The Response Synthesis module abstracts away context window limitations, while the Document Metadata Management feature allows deep customization of metadata, potentially boosting retrieval performance. Blogpost, Tweet
11. LlamaIndex introduces Recursive Retrieval, a concept that utilizes the hierarchical nature of knowledge. A Node in LlamaIndex can contain references to other retrievers or query engines. This process starts with a retriever and recursively explores links to others. For instance, structured tables from a PDF can be extracted, each represented as a data frame. These tables can be referenced by IndexNode objects embedded with other Nodes. During a query, if an IndexNode is among the top-k nodes, it triggers another retriever or query engine, allowing sophisticated querying overall data. Docs, Tweet
12. LlamaIndex introduces OpenAI agent streaming for efficient function calling and enhances user experience by providing progress bars during index creation for a real-time understanding of the process duration. Tweet
13. LlamaIndex introduces personalized data interaction through system prompts, callback events for SubQuestionQueryEngine, and a streamlined process for Azure OpenAI integration. Docs_AOI, Notebook_personality, Tweet
14. LlamaIndex leverages LLM’s to automatically extract metadata, significantly enhancing the relevance and precision of information retrieval. This is achieved through five key MetadataExtractor modules (SummaryExtractor, QuestionsAnsweredExtractor, TitleExtractor, MetadataFeatureExtractor) that augment text with rich, context-specific details. Docs, Tweet

Tutorials:

1. Anyscale tutorial on “How to Build an LLM Query Engine in 10 Minutes using LlamaIndex.”
2. Erika Cardenas tutorial on how to load data into Weaviate and how to connect LlamaIndex to a Weaviate instance using LlamaIndex.
3. Wenqi Glantz tutorial on Refreshing Private Data Sources with LlamaIndex Document Management.
4. Michael Hunger tutorial on Load in data from neo4j, NebulaGraph, and index/query with LlamaIndex using GraphDB Cypher and GraphQL data loaders.
5. Pradip Nichite video tutorial and blogpost on Mastering LlamaIndex: Create, Save & Load Indexes, Customize LLMs, Prompts & Embeddings.

Webinars And Podcasts:

1. Webinar on Graph Databases, Knowledge Graphs, and RAG with Wey (NebulaGraph).
2. Webinar with Albus — a comprehensive Slackbot for enterprise search, xpress.ai — a low-code solution for building LLM workflows + agents and ImmigrantFirst.ai — assistant to help immigrants complete their EB-1A/O1 apps more efficiently.
3. Data Exchange Podcast with Ben Lorica on LlamaIndex

Events:

Ravi Theja gave talks on “LlamaIndex: Basics To Production” at Accel Partners and Together VC Fund in India.