AI in Business Intelligence: ChatTransform Overview (EN)

In an AI supported, emerged business intelligence practice, the importance of prompt engineering cannot be overstated. The ability to automatically analyse data using AI tools holds immense potential for extracting valuable insights and driving data-informed decision-making. In this blog post, we will explore the fascinating world of AI-powered data analysis through the lens of ChatTransform, a cutting-edge BI tool. We will dive into the successes, challenges, and future prospects of harnessing the power of prompt engineering in BI.

Recently, we conducted an intriguing experiment to test the capabilities of our first use case at AI-Garage, ChatTransform. With a simple text input from the user and a dataset as its foundation, ChatTransform effortlessly transforms the data and provides the desired analysis. The structured dataset used contained the following six columns:

The "day" column represents the date & time period associated with the entry. The "article" column indicates the specific item or product being referred to in the data. The "channel" column represent the marketing or distribution channel through which the product is being sold or promoted. The “name” column represents article name. The "brand" column represents the brand or company associated with the product or service being analysed. Lastly, the "revenue" column contained numerical values representing the generated income corresponding to each data entry. We designed an exhaustive set of 100 test cases to evaluate system’s performance. During our testing, we explored the diverse range of abilities within ChatTransform.

The results were a mix of successes and failures, showcasing both the promise and the areas for improvement in AI-driven data analysis. In addition to its Data Analytical abilities, ChatTransform also demonstrated its flair for convoluted, long input prompts using its Natural Language Processing (NLP) capabilities. We pushed its boundaries by presenting it with creatively framed queries, challenging its ability to understand the nuances. Remarkably, ChatTransform rose to the occasion, providing insightful responses that showcased its adaptability and potential to handle unconventional scenarios.

Let's delve into the different categories we encountered in our experiment.

Granularity Matters

In some cases, AI misinterpreted the intended granularity of the analysis. Instead of providing results at a specific level, it presented outputs considering the entire dataset as a whole. For example when prompted with, "Plot the sales volume over time for any 1 brand”. Even though the input is specifically asking the system to choose any one specific brand, it creates a plot across all brands. This can be interpreted as a lack of understanding or perhaps system’s avoidance to make decisions in an unsupervised setting. With further refinement, ChatTransform has the potential to deliver targeted and granular insights, empowering businesses to make more informed decisions.

Hallucination and the Unexpected

When asked to analyse parameters that did not exist in the dataset, we encountered an intriguing aspect that seems to be a common tendency AI's generative models exhibit, "hallucination”. Surprisingly, AI would sometimes mistake a different parameter for the non-existent one, resulting in unexpected outputs. This pattern was often seen with ‘revenue’ parameter in our dataset. When asked to analyse data on ‘price’ of items, the system used ‘revenue’ parameter as a substitute, even though they are utterly different in both, semantic and syntactic terms. Although this behaviour highlighted the need for improved parameter recognition, it also showcased the AI's ability to identify patterns and make connections within the data. This trait of the tool led to positive unexpected outcomes as well. When provided with prompts containing general everyday words instead of data-specific terms, ChatTransform demonstrated its ability to decipher the intended context and accurately identify the relevant columns from the dataset. For example: “Which is the most sold product?”. This feature showcases the tool's adaptability and linguistic understanding, enabling users to extract insights even when they express their queries in more informal or non-technical language. By bridging the gap between technical and everyday language, ChatTransform enhances accessibility and empowers users to effortlessly interact with their data. This opens exciting possibilities for discovering hidden insights and uncovering unexpected relationships.

The Importance of Preprocessing

The Importance of Preprocessing: Our experiment also shed light on the importance of proper preprocessing. Some errors surfaced due to the AI's inability to handle different data types effectively. For instance, columns with string data encoded as integers (‘brand’ column in our dataset was one such attribute) or columns containing null values (‘name’ column mostly had null data entries) caused errors and yielded "none type object is not iterable" messages on the frontend. This highlighted the necessity of thorough preprocessing techniques to handle diverse data formats and ensure accurate analysis. Nonetheless, this obstacle can be overcome with improved data preparation using refined prompts. Since this problem was identified during testing, latest version of chatTransform has handled this case by adding appropriate prefix to the system query, to facilitate basic data pre-processing is carried out.

The Strength of Statistical Analysis

Amidst the challenges, a varied range of standard statistical analyses performed exceptionally well. Prompts on the lines of, "Calculate the standard deviation of revenues”, “Draw a quarterly revenue by channel using whichever plot you deem fit for such representation”, "Calculate the percentage change in revenue from the previous month." all exhibit expected results. The logically sound algorithms employed by ChatTransform consistently produced reliable and insightful results. This demonstrated the AI's ability to handle statistical calculations, enabling businesses to gain valuable insights into trends, patterns, and correlations within their data. These statistical capabilities hold immense potential for driving evidence-based decision-making in various industries.

AI-Garage’s first experiment with ChatTransform showcased both the remarkable accomplishments and the areas for further development in AI-powered data analysis for BI. At the same time, it is crucial to acknowledge that human oversight and expertise still remain indispensable in this domain. While AI tools like ChatTransform showcase remarkable capabilities, they cannot yet fully replace the critical thinking and industry knowledge possessed by data professionals. AI systems, no matter how advanced will still have to be viewed from a human-centric vantage point. Instead, the role of data professionals will evolve from routine tasks such as creating visuals to focusing on supervising the data quality; taking up the burden of ensuring high-quality data inputs and enhancing the performance of AI models. Data professionals will only be equipped furthermore with AI systems as their side-kick. Inspiring further research on how AI can contribute to data modelling. Additionally, it is all the more essential to familiarise ourselves with the distinctive behaviour of Language Models, including the potential for hallucinations, an unseen battle which we did not get tangled up in, with traditional BI approaches. By embracing these considerations, we can harness the power of AI in Business Intelligence while maintaining a vigilant and well-informed approach. The potential of prompt engineering in guiding and enhancing AI tools stands to be undeniable. With ongoing advancements in AI algorithms, enhanced preprocessing techniques, and improved parameter recognition, the future of AI in BI looks exceedingly promising.