AILab Howest

Embedding models

A computer does not perceive meaning in text like a human does, but it can perform calculations with numbers at high speeds. Aembedderis aAI modelturn that text into a series of numbers that we avectornames. These vector representations have two important properties:

1. They contain information that says something about therelationships between words, but it can be demonstrated using vectors that "brunch" in terms of meaning lies somewhere between "breakfast" and "lunch." Embedders also assign similar numbers to synonyms.
2. We can there thestrength of similarityexpress between two words. There are mathematical calculations behind this such as the “cosine similarity.

There are many open-source embedding models available onHuggingFaceinOllamaDuring the workshop, we got to work withEmbeddingGemmaa recent development from Google.

RAG

LLMs are in terms oflimited knowledgeto their training dataset. They can alsohallucinatingand are they sometimesbiased.

Retrieval Augmented GenerationRAG is an implementation of LLMs where we can query our own knowledge database. The chatbot can therefore consult and mention sources in its response. This largely avoids the problems with LLMs, but RAG can also make mistakes.

Such aknowledge databasetypically consists of your business documents, or documentation of machines in an industrial context. These are oftenconfidentialof containsensitive personal data.

To guarantee privacy, we can also host the entire RAG pipeline locally ourselves. There are open-source "all-in-one" platforms such asRAGFlowthe man withDockercan run on any computer. During the workshop, we manually worked out RAG withLangChainin Python.

If privacy is not a priority, there are also cloud alternatives such asNotebookLMfrom Google.

Agents

An agent is an LLM thatcan perform actions, these are some examples:

• Querying a knowledge database (RAG)
• Fetching the weather report
• Sending an email in your name
• …

In the context of agents, we refer to these actions as "toolsTo make the LLM aware of the available tools, we use theMCP protocol.

Many apps now also offer aMCP serveryes, this contains the tools with which an LLM can perform actions on that app. A developer or platform such asn8ncan use this to build an agent. During the workshop, we created our ownlocal MCP serverset up withFastMCPin Python.

Guardrailing

LLMs and agents are vulnerable tomalicious prompts(prompt injection of jailbreaking). Also, unwanted outputs such as profanity, personal data, or incorrect languages can be problematic. Therefore, areextra security measuresessential in production environments.

During the workshop, we usedGuardrails AIa Python library that contains pre-made guardrails. You can use it to pre- or post-process user prompts and LLM responses and perform checks on them. Underlying this, they use various techniques, one of which is the use offine-tuned language modelsspecialized in guardrailing tasks (e.g.Llama Guard 3Another open-source alternative isGranite Guardianfrom IBM.

Infrastructure

During the workshop, we also delved deeper into the infrastructure requirements to run AI applications. In many cases, you need powerful NVIDIA GPUs. Through subsidies via the Interreg Flanders-Netherlands Art-IE project & a VLAIO infrastructure call, Howest has purchased a new AI server with the specifications below:

• CPU:2x Intel Xeon Platinum 8562Y+ (128 threads)
• GPU:8x NVIDIA H200 SMX (141GB VRAM)
• RAM:2TB DDR5
• Storage:2x Dell NVMe 7500 (3.84TB) + 5x Micron 9300 (3.5TB)

It is important that there is quite a bithidden costsadditional costs such as electricity, licenses & software, support, …

In terms of software, we useKubernetesin combination withKubeflowThis is aopen-source ecosystemthat integrates well with the hardware & software ofNVIDIAKubeflow primarily consists of a modularcentral dashboardYou can add modules to enable the creation of Jupyter Notebooks, pipelines, model training, etc., and manage their resources.