Technology and availability of generative AI and especially large language models (LLMs) have taken a leap in recent years. Large parties like OpenAI, Microsoft and Google dominate this market. It is also possible to run LLMs locally. Challenges might be hardware requirements and the required knowledge to get started. When choosing the right tools this is less difficult than you might expect. I suggest getting started with LM Studio as GUI and ollama as CLI tool for running models locally. In this blog post I'll try to get you started quickly and provide some essential background information ..read more

To quote Wikipedia (here): "Stable Diffusion is a deep learning, text-to-image model released in 2022. It is primarily used to generate detailed images conditioned on text descriptions, though it can also be applied to other tasks such as inpainting, outpainting, and generating image-to-image translations guided by a text prompt.". This of course sounds nice, but what makes it special and how can you use it? Why is Stable Diffusion special? The dataset The Stable Diffusion model was trained on three subsets of LAION-5B: laion2B-en, laion-high-resolution, and laion-aesthetics v2 5+. These dat ..read more

When using AI models like Stable Diffusion, sometimes input images need to be of a specific size. In case of Stable Diffusion, multiples of 64 are required. Stable Diffusion (at least 1.5) works best with images of 512 pixels in width or height. If you for example take an image of 599 x 205 pixels and you resize it to 1496 x 512 (maintaining the aspect ratio), you end up with 1496, which is not a multiple of 64. In order to obtain a usable image, it needs to be padded to a size of 1536 x 512 to allow processing. In order to batch resize and pad images I created a Python script using OpenCV. Wh ..read more

In the IoT world, some devices generate large volumes of events that can be difficult for back-end systems to process in real time. Of course you can use NiFi to throttle messages. However, this will not be sufficient if the flow of events is consistently higher than what can be handled by the back-end system. A way to deal with this is to let Apache NiFi group and filter messages based on a specific attribute and only letting through the latest message for a specific device, in a certain timeframe. In this blog post I'll illustrate how you can do this. The trick is to merge several messages t ..read more

This is not a usual topic for this blog but I wanted to share this anyway so people can refer to it and use it. Due to the current challenges on the energy market, prices for both gas and electricity in the Netherlands and probably Europe as a whole have risen dramatically. The prices are currently so high large groups of people cannot pay their gas and electricity bills anymore. In this blog post I'll give some suggestions for things which can help you reduce your gas and electricity bill. Some of these are specific to the Netherlands.  What I'm trying to do is to optimize my gas/energy ..read more

There are several cases where you might want to use Azure Log Analytics to monitor your NiFi instances. An obvious one is when NiFi is running in Azure. Azure Log Analytics can also be used as single monitoring/alerting solution for multiple applications making operations easier by providing a single interface for this. You might want this if you want to monitor business processes which span multiple applications and you want to monitor the entire process to for example identify bottlenecks. In this blog post I'll show you how easy it is to achieve this using the AzureLogAnalyticsReportingTask ..read more

When you import a new process group or upgrade an existing one, missing parameters contexts and parameters will automatically be added. The new parameters will be filled with values from the environment where the process group was committed to the Registry (except sensitive parameter values). This is usually a development environment. NiFi 1.15 adds parameter context inheritance. If however you are on a lower version and have many similar process groups, you can have many copies of parameter contexts. If you add a large number of flows and need to add a large number of parameter contexts, it c ..read more

Jolt is a Java library which can be used to transform JSON to JSON (here). A Jolt transformation itself is also a JSON file. You can use it in products such as Apache NiFi and Apache Camel. In this blog post I'll describe my first experiences with Jolt transformations.  For me personally Jolt transformations are not intuitive and not that powerful when for example compared to the capabilities of XSLT for transforming XML. It is available in Apache NiFi though and can be used without the 'execute code' permission, that is why I decided to use it to transform JSON files. I might misundersta ..read more

Apache NiFi is a powerful open source integration product. A challenge you might encounter when integrating systems is that one system can produce JSON messages and the other has a SOAP API available. In this blog post I'll show how you can use NiFi to convert JSON input to a SOAP service call. This involves abstracting an AVRO schema for the JSON, converting it to XML and transforming the XML to a SOAP message.  In this example I'm using several publicly available websites. You should of course be careful. Do not copy/paste sensitive XML or JSON on these sites! Input I've used the foll ..read more

Apache NiFi has a powerful web-based interface which provides a seamless experience between design, control, feedback, and monitoring. Sometimes however, you want to automate tasks instead of doing them manually using the UI. This does not only allow you to perform the tasks a lot quicker but it also helps make them more reproducible. It allows you to incorporate tasks in for example a CI/CD system without requiring human intervention. A NiFi feature to help you automate tasks is its powerful API. In order to more easily use this API from Python, NiPyAPI is available. In this blog post I'll de ..read more