TLDR; High performance data teams transform their organisation by making sure the right people see the right data at the right time to make the right decisions. The recipe for an effective in-house data team begins with strong C-suite sponsorship. You align organizational goals, adopt a product mindset and position your team as an innovation engine. You harness modern cloud native data platforms such as Microsoft Fabric to deliver value rapidly and safely, establishing DataOps from day one. The team is a "centre of enablement", democratising data by leading a data skills transformation across ..read more

Boosting Apache Spark Performance with Small JSON Files in Microsoft Fabric. Learn how to achieve a 10x performance improvement when ingesting small JSON files in Apache Spark hosted on Microsoft Fabric. Ian Griffiths, Technical Fellow at endjin, shares insights and techniques to overcome Spark's challenges with numerous small files, including parallelizing file discovery and optimizing data loading. Follow along for detailed steps and tips to significantly enhance your Spark data processing workflows using Apache Spark in Microsoft Fabric. 00:00 Introduction to Performance Improvement in Apa ..read more

TLDR; FileSystemWatcher has a classic .NET event handler based API. This can make it difficult to write compensating logic to deal with some of the idiosyncratic behaviours that it (and the underlying file system) can exhibit. Thankfully the Reactive Extensions for .NET (Rx .NET) provides powerful APIs which can convert these events into an observable event stream which can easily be filtered and manipulated to provide a more modern, high-level and easy to work with API. Note: FileSystemWatcher has a number of known issues and (cross platform) limitations. You should be aware of these before ..read more

In this series we are cataloging common patterns with JSON Schema and the .NET code generated by Corvus.JsonSchema. It is especially useful for developers who want to move to a schema-first approach, but are unsure how that will map to their .NET code. We are focusing on draft 2020-12 and the dotnet8.0 code generation, but very similar patterns apply for older versions of both .NET and draft 2019-09, draft 7, and even draft 6. We will highlight the key differences as we go. If you have no experience of JSON Schema at all, I would recommend you read the getting started step-by-step documentatio ..read more

In this series we are cataloging common patterns with JSON Schema and the .NET code generated by Corvus.JsonSchema. It is especially useful for developers who want to move to a schema-first approach, but are unsure how that will map to their .NET code. We are focusing on draft 2020-12 and the dotnet8.0 code generation, but very similar patterns apply for older versions of both .NET and draft 2019-09, draft 7, and even draft 6. We will highlight the key differences as we go. If you have no experience of JSON Schema at all, I would recommend you read the getting started step-by-step documentatio ..read more

In this series we are cataloging common patterns with JSON Schema and the .NET code generated by Corvus.JsonSchema. It is especially useful for developers who want to move to a schema-first approach, but are unsure how that will map to their .NET code. We are focusing on draft 2020-12 and the dotnet8.0 code generation, but very similar patterns apply for older versions of both .NET and draft 2019-09, draft 7, and even draft 6. We will highlight the key differences as we go. If you have no experience of JSON Schema at all, I would recommend you read the getting started step-by-step documentatio ..read more

In this series we are cataloging common patterns with JSON Schema and the .NET code generated by Corvus.JsonSchema. It is especially useful for developers who want to move to a schema-first approach, but are unsure how that will map to their .NET code. We are focusing on draft 2020-12 and the dotnet8.0 code generation, but very similar patterns apply for older versions of both .NET and draft 2019-09, draft 7, and even draft 6. We will highlight the key differences as we go. If you have no experience of JSON Schema at all, I would recommend you read the getting started step-by-step documentatio ..read more

In this series we are cataloging common patterns with JSON Schema and the .NET code generated by Corvus.JsonSchema. It is especially useful for developers who want to move to a schema-first approach, but are unsure how that will map to their .NET code. We are focusing on draft 2020-12 and the dotnet8.0 code generation, but very similar patterns apply for older versions of both .NET and draft 2019-09, draft 7, and even draft 6. We will highlight the key differences as we go. If you have no experience of JSON Schema at all, I would recommend you read the getting started step-by-step documentatio ..read more

In this series we are cataloging common patterns with JSON Schema and the .NET code generated by Corvus.JsonSchema. It is especially useful for developers who want to move to a schema-first approach, but are unsure how that will map to their .NET code. We are focusing on draft 2020-12 and the dotnet8.0 code generation, but very similar patterns apply for older versions of both .NET and draft 2019-09, draft 7, and even draft 6. We will highlight the key differences as we go. If you have no experience of JSON Schema at all, I would recommend you read the getting started step-by-step documentatio ..read more

In this series we are cataloging common patterns with JSON Schema and the .NET code generated by Corvus.JsonSchema. It is especially useful for developers who want to move to a schema-first approach, but are unsure how that will map to their .NET code. We are focusing on draft 2020-12 and the dotnet8.0 code generation, but very similar patterns apply for older versions of both .NET and draft 2019-09, draft 7, and even draft 6. We will highlight the key differences as we go. If you have no experience of JSON Schema at all, I would recommend you read the getting started step-by-step documentatio ..read more