Streaming analytics refers to the process of analyzing real-time data that is generated continuously and rapidly from various sources, such as sensors, applications, social media, and other internet-connected devices. Streaming analytics platforms enable organizations to extract business value from data in motion, similar to how traditional analytics tools derive insights from data at rest. DeltaStream is a unified serverless stream processing platform to manage, secure and process all your event streams and is based on Apache Flink. Hojjat Jafarpour is the Founder and CEO at DeltaStream and ..read more

Distributed databases are necessary for storing and managing data across multiple nodes in a network. They provide scalability, fault tolerance, improved performance, and cost savings. By distributing data across nodes, they allow for efficient processing of large amounts of data and redundancy against failures. They can also be used to store data across multiple locations for faster access and better performance. Turso is an edge-hosted, distributed database based on libSQL, an open-source and open-contribution fork of SQLite. It was designed to minimize query latency for applications where q ..read more

“DataSet” is a log analytics platform provided by Sentinel One that helps DevOps, IT engineering, and security teams get answers from their data across all time periods, both live streaming and historical. It’s powered by a unique architecture that uses a massively parallel query engine to provide actionable insights from the data available. John Hart is a distinguished engineer leading the Event DB team, where he’s responsible for the time series database that powers the Dataset product. John is our guest here today. Full disclosure: SentinelOne is a sponsor of Software Engineering Daily. Spo ..read more

There are many types of early stage funding available from friends and family to seed to series A.  Some firms invest across a wide set of technologies and seek only to provide capital. Others are in it for the long haul – they focus on specific areas of technology and develop both long term relationships and deep expertise over time.   Today, we are interviewing Matt Turck of First Mark Capital, who is in it for the long haul and whose portfolio companies include Dataiku, Crossbeam, Ada, Cockroach Labs, Clickhouse and more.  Today we will talk about Matt’s career, investme ..read more

Uber is one of many examples we’ve discussed on this show that has changed the world with big data analysis. With over 8 million users, 1 billion Uber trips and people driving for Uber in over 400 cities and 66 countries, Uber has redefined an entire industry in a very short time frame. It’s difficult to find precise details about Uber’s big data infrastructure online, but we know they collect every possible data point about their drivers and riders. Matching riders and drivers, setting ride fares, predicting demand for cars – these are some examples of what Uber does with its data. In this e ..read more

The quantity and quality of a company’s data can mean the difference between a major success or major failure. Companies like Google have used big data from its earliest days to steer their product suite in the direction consumers need. Other companies, like Apple, didn’t always use big data analytics to drive product design, but they do now.  The company Axiom has created a large suite of advanced browser robots that perform difficult tasks like consolidating data across many web applications, extracting data from public sites or from behind logins, data entry, user interface automation ..read more

The Presto/Trino project makes distributed querying easier across a variety of data sources. As the need for machine learning and other high volume data applications has increased, the need for support, tooling, and cloud infrastructure for Presto/Trino has increased with it. Starburst helps your teams run fast queries on any data source. With Starburst you get a single point of access to your data, no matter where it’s stored and it supports high concurrency. Whether it’s fast SQL queries on your data lake or faster queries across multiple datasets, Starburst helps your teams run analytics an ..read more

Building and managing data-intensive applications has traditionally been costly and complex, and has placed an operational burden on developers to maintain as their organization scales. Todays’ developers, data scientists, and data engineers need a streamlined, single cloud data platform for building applications, pipelines, and machine learning models — without having to move or copy their data. Platforms like the Snowflake Data Cloud provides a unified tool for developers to easily build data applications with Python using Streamlit’s open source framework and Snowflake’s Native Application ..read more

Originally published on April 12, 2022. As companies move to Spark and a Lakehouse architecture, they are realizing that the data tools are lagging way behind.  You need to be a programmer to effectively use Spark and Airflow. There are some low-code ETL tools, but is that enough?  Companies want to treat their data pipelines like mission-critical apps.  They want DevOps for data, with GIT, test coverage, and CI/CD.  In addition, companies need end-to-end visibility of their data pipelines, including monitoring, metadata, and column-level lineage. Where are the data tools f ..read more

Data is becoming a bank’s biggest asset. These complex enterprises have a huge opportunity ahead – to transform themselves to become a trusted hub of a much broader data ecosystem that goes beyond the financial industry and helps to form a new class of cross-industry experience architectures that are scalable and transparent. The data physics that is needed for such emerging systems runs on consent and privacy preservation rather than black-boxed data lakes. A foundation for making this happen lies in the ability to use distributed, heterogenous data effectively and transform it into experien ..read more