Introduction

Artificial Intelligence (AI), especially Large Language Models (LLM), represent a technological breakthrough that has changed how we interact with information and digital content. These LLM models are usually pre-trained on huge datasets, where the most famous are the currently large-scale projects of OpenAI, Google and Anthropic built as GPT (Generative Pre-training Transformer). Intensive pre-training and tuning techniques (e.g. via PEFT, LORA and RLHF) allow given LLMs to generate a range of new artifacts. The generated texts, images, videos and many more can be surprisingly coherent and relevant depending on the quality of the input from the "Prompt engineers" as far as the skilled user is concerned.

Within the aforementioned workshop, we presented the following five separate and distinct Prompt Engineering related topics as part of a broader team of researchers as multiple different experts from different companies, which we present in the summary agenda at the end.

1. Principles and methods of GenAI & LLMs
2. Techniques and Art of Prompt Engineering
3. GenAI for image and video generation
4. ML & data analytics using LLMs
5. Complex solutions with ChatBots & more

1. Principles and methods of GenAI & LLMs

The basis of LLMs today is most commonly an architecture called Transformer, which was introduced in 2017 by Google in their seminal 2017 paper titled "Attention Is All You Need". In that paper, the company showed a Transformer model that dispenses with the recurrent layers commonly used in previous models for tasks such as machine translation. Transformer has since become the underlying architecture for many other developments in Natural Language Processing (NLP), including models such as GPT and BERT (Bidirectional Encoder Representations from Transformers). This architecture now allows models to process and generate text with a high level of contextual understanding, thanks to mechanisms such as self-attenuation and positional encoding.

Image 1: The path from RNN to Tranformer architecture

2. Techniques and Art of Prompt Engineering

There are many well-known prompting techniques such as setting Digital Persona, context, delimiters according to Markdown syntax and others listed below under visualization. Today there are even special web marketplaces where the best prompts can be bought. Prompting, however, can be compared to the art of communication between people in a more general sense. That is, you need to know what you want to say, phrase it well with respect to your interlocutor, so that you get a result that fits your specific task.

Below is a summary of techniques in the form of a visual, as well as a list of good resources where best practices in prompt engineering can be adopted:

• Handbook #1 Prompt Engineering (Detailed)
• Handbook #2 Prompt Engineering (OpenAI)
• Handbook #3 on Prompt Engineering (Guide)

Image 2: Prompting techniques are not a panacea, you need to be able to "talk" to LLMs

3. GenAI for image and video generation

We all have the same tools available for GenAI and sometimes we have an artistic creation in a few clicks and sometimes we just overwhelm the infosphere with "choppy videos" that can't articulate in Czech and in other non-English languages.

During the workshop we introduced the following tools suitable for generating images and videos, including their limits and specifics.

• Image Generator #1 Midjourney
• Image Generator #2 Dalle
• Image Generator #3 Adobe Firefly
• Video generator #A Pika
• Video generator #B Synthesia

Image 3: Anyone can generate images, but it's not always easy to watch

4. ML & data analytics using LLMs

LLMs can save data scientists and computer science students a lot of work in dealing with the sub-steps associated with routine work and data pre-processing or interpretation. For example, it is not a problem to insert a CSV file into ChatGPT and ask for formatting modifications or to perform the whole ETL (Extract-Transform-Load) procedure by incremental instructions.

An interesting option is to start data operations using LLM instructions entered in a common language, but if we reach the point where it is more efficient to continue as a programmer in a specific language, it is not a problem to be shown Python code that we can further edit ourselves in Jupyter notebooks or some IDE.

In contrast, apart from the ability to use LLM to process numerical tasks, which has its limitations, LLM often excels and outperforms the conventional data scientist in processing text strings and language in general. This is not only for the common tasks of identifying the sentiment of text writers, but also for more complex processing such as the fake news domain. The workshop gave an example of analyzing the actual case of the Slovak presidential election in 2024. We used ChatGPT to uncover patterns of manipulative behavior as well as to compare facts from independent sources and misleading news stories of one particular presidential candidate. LLM results were significantly better than other specific data science tools applied to the same task.

Image 4: LLM often makes data easier to work with and excels at word processing

5. Complex solutions with ChatBots & more

LLMs have revolutionized the capabilities of chatbots, workflow management and automation, enabling more complex solutions across industries. Chatbots using LLMs can handle complex customer queries with high accuracy and human-like response speed, greatly improving user experience and operational efficiency.

In workflow automation, these models help streamline business processes by intelligently automating routine tasks and providing decision support, freeing up human resources for other activities. In addition, integrating LLM with advanced automation tools often leads to innovative applications in specific industries such as finance for real-time risk assessment, invoice processing or legal services for document analysis. These solutions not only increase productivity but also push the boundaries of what automated systems can achieve, transforming business operations and customer interactions.

Most companies today build ChatBots on top of OpenAI's ChatGPT product, which is more suited for general conversation rather than industry-specific solutions with unique value-adds and answers. This is because to design an industry-specific LLM solution over open source solutions like Llama 2 or Mistral would mean additional LLM training, specific data, and very often chain-of-models and automation to solve a complex task for which companies lack the knowledge and data.

Image 5: Building complex solutions and Chatbots can be done either well or quickly

Conclusion

This article was written and the images were generated using LLMs jointly by Richard A. Novak and Jiří Korčák from Omnicrane as a summary of the conference organized on April 10, 2024 by Prague Data Ethics Lab and Omnicrane. The event also included contributions from colleagues from PwC, V# Venture Studio and Silesian University Opava. The titles and order of the conference presentations with the assignment of each author to a topic are shown in the figure below, which formed the original invitation to the conference.