Streaming data has emerged as a critical component of modern data strategies. Streaming data refers to data that is continuously generated by various sources, these typically send data records in small sizes (such as bytes or kilobytes). This blog explores what streaming data is, why it’s important, its applications, and the tools used to manage it. What is Streaming Data? Various sources continuously produce streaming data and send it in real-time to an IT system for processing and analysis. Unlike batch data processing, which stores data sets for later processing, streaming data aims for imm ..read more

Medallion architecture, a term increasingly recognized within the realms of data management and software development. It presents a compelling framework for building scalable, flexible, and efficient data platforms. This blog post explores the core concepts of medallion architecture, its benefits, and real-world applications. It aims to shed light on why it has gained popularity among data engineers and architects. What is Medallion Architecture? At its heart, medallion architecture is a data modelling strategy designed to streamline the process of data ingestion, storage, processing, and anal ..read more

Following on from the blog last week I thought I’d write a similar guide to extracting data from Joblogic. Joblogic is a popular Field Service Management (FSM) software that helps facility management companies run their operations. FSM software is particularly useful for companies that provide services at the customer’s location. These companies are generally involved in repairs, maintenance, installations, or any on-site services. Joblogic operates in a highly competitive market competing with the likes of ServiceMax, IFS and BigChange. The challenge is how to produce meaningful actions ..read more

Having worked with a few customers on automating the data output from their HR system, namely BambooHR. I thought I’d summarise some some of the considerations on extracting data from BambooHR. BambooHR is a cloud-based Human Resources Information System (HRIS) designed to simplify HR processes for businesses. It’s known for its user-friendly interface, scalability, and the ability to streamline complex HR tasks. As with any system the data collected is only as useful as the actions from it. Whilst BambooHR can improve HR analytics, reporting, and decision-making processes in your organization ..read more

This post explains how to sync slicers in Power BI, without any complex code Slicers are an excellent way to give your users the ability to explore data, allowing users to use the slicers to filter the data down to particular information of interest. But, if your data has different data sets with common values e.g. a week number, you would need to either 1. Create a relationship between the data sets  or 2. Enable Sync Slicers to keep different data sets aligned In our example here we will show how you can use the sync slicers across three different data sets 1. A Sales Budget 2. A We ..read more

Having recently stumbled on User-Defined Types (UDTs) in SQL Server. I thought I’d write a blog to explain what user-defined types are? Why you should you use them? This blog post delves into what UDTs are, why they are useful, and how to use them in SQL Server. What are User-Defined Types (UDTs)? In SQL Server, User-Defined Type (UDT) are a custom data type that you can create to extend the built-in data types. UDTs allow you to define your own data structures with specific properties and behaviours. They are created based on the system data types and can include multiple elements and methods ..read more

On a recent project I have been looking at whether to use Azure Logic Apps as an alternative to Azure Data Factory (ADF). This brought me to compare these 2 technologies. This blog aims to summarise key differences between Azure Data Factory vs Logic Apps. It also aims to discuss the cost implications of Azure data factory pricing as well as that of Azure logic apps. Both technologies are design to automate workflows and integrate systems. They are powerful, scalable solutions within the Azure ecosystem. Despite their similarities in functionality, they cater to different needs and scenarios ..read more

Goolge Search Console is a very powerful tool to measure SEO performance, this post highlights how with a little effort you can use Google Search Console Data in Power BI to create much more compelling dashboards It tracks a vast amount of really insightful data, but the reporting is limited The example below is the data for our website over the 12 months ending 29th February 2024 The most disappointing aspect to this is that the dates are fixed to individual dates, with no options for aggregation. This results in a very noisy report. Similar to many business to business organisations we ..read more

This quick guide shows you how to split a column in BigQuery When working with text data, quite often you need to split a column or trim excess characters to clean the data This quick guide shows you how to it using URL data from Google Analytics GA4 to create separate columns for the Clean URL and the URL parameter   Initial Set up Lets start with a list of URLs from our web traffic You can copy this code to try it   WITH URLs AS (SELECT 'https://www.selectdistinct.co.uk/?utm_campaign=directory&utm_medium=referral&utm_source=themanifest.com' as page_location UNION ALL ..read more

Importing XML data into Excel allows you to have a readily available data source that can be refreshed on Demand Once that data loaded into Excel you can use Excels features to explore, filter, sort or augment that data from other sources What is an XML file? An XML file is a plain text file that uses custom tags to describe the structure and features of the document. It can be opened and edited with text editors, web browsers, online editors, and application-specific programs. XML files are often used to store and exchange data between different systems or applications. For example, XML files ..read more