Streaming and real-time data has high business value, but that value can rapidly decay if not processed quickly. If the value of the data is not realized in a certain window of time, its value is lost and the decision or action that was needed as a result never occurs. Streaming data – whether from sensors, devices, applications, or events – needs special attention because a sudden price change, a critical threshold met, a sensor reading changing rapidly, or a blip in a log file can all be of immense value, but only if the alert is in time.
In this webinar, we will review the landscape of streaming data and message queueing technology and introduce and demonstrate a method for an organization to assess and benchmark—for their own current and future uses and workloads—the technologies currently available. We will also reveal the results of our own execution of the OpenMessaging benchmark on workloads for two of the platforms: Apache Kafka and Apache Pulsar.
What Will Be Discussed:
The Evolution of Queuing, Messaging, and Streaming
Today’s Technology Landscape
Assessing Performance: The OpenMessaging Benchmark
Considerations for Your Evaluation
Register now and join Gigaom Research and our sponsor Streamlio for this free expert webinar.
Who Should Attend:
CIOs, purchasers and recommenders of data platforms
This free 1-hour webinar from GigaOm Research brings together leading minds in cloud data analytics, featuring GigaOm analyst Andrew Brust, joined by guests from cloud big data platform pioneer Qubole and cloud data warehouse juggernaut Snowflake Computing. The roundtable discussion will focus on enabling Enterprise ML and AI by bringing together data from different platforms, with efficiency and common sense.
In this 1-hour webinar, you will discover:
How the elasticity and storage economics of the cloud have made AI, ML and data analytics on high-volume data feasible, using a variety of technologies.
That the key to success in this new world of analytics is integrating platforms, so they can work together and share data
How this enables building accurate, business-critical machine leaning models and produces the data-driven insights that customers need and the industry has promised
You’ll learn how to make the lake, the warehouse, ML and AI technologies and the cloud work together, technically and strategically.
Register now to join GigaOm Research, Qubole and Snowflake for this free expert webinar.
Many conversations around GDPR seem to follow a similar sequence as Dave Lister’s experience in the opening episode of Red Dwarf.
Holly: They’re all dead. Everybody’s dead, Dave.
Lister: Peterson isn’t, is he?
Holly: Everybody’s dead, Dave!
Lister: Not Chen!
Holly: Gordon Bennett! Yes, Chen. Everyone. Everybody’s dead, Dave!
Holly: He’s dead, Dave. Everybody is dead. Everybody is dead, Dave.
Lister: Wait. Are you trying to tell me everybody’s dead?
So, yes, GDPR affects all kinds of data. Big data, small data, structured and unstructured data, online and offline, backup and archive, open or grey, digital or paper-based data. It’s all data, and therefore GDPR applies to it.
This simultaneously makes the task of GDPR compliance very easy, and very difficult. Easy, because decision makers don’t have to worry about what data is involved. And very difficult, because few organizations have a clear handle on what data is stored where. That filing cabinet in the back of a warehouse, the stack of old tapes on top of a cupboard, that rack of servers which were turned off… yeah, all of them.
Because that’s not the focus of GDPR, you know, the technology gubbins, complexity and all that. The regulation quite deliberately focuses on personally identifiable information and its potential impact on people, rather than worrying about the particular ramifications of this or that historical solution, process or lack of one.
At the same time, this does suggest quite a challenge. “But I don’t know what I have!” is a fair response, even if it is tinged with an element panic. Here’s some other good news however — laws around data protection, discovery, disclosure and so on never distinguished between the media upon which data was stored, nor its location.
You were always liable, and still are. The difference is that we now have a more consistent framework (which means less loopholes), a likelihood of stronger enforcement and indeed, potentially bigger fines. To whit, one conversation I had with a local business: “So, this is all stuff we should have been doing anyway?” Indeed.
Of course, this doesn’t make it any easier. It is unsurprising that technology companies and consulting firms, legal advisors and other third parties are lining up to help us all deal with the situation: supply is created by, and is doing its level best to catalyse, demand. Search and information management tools vendors are making hay, and frankly, rightly so if they solve a problem.
If I had one criticism however, it is that standard IT vendor and consulting trick of only asking the questions they can answer. When you have a hammer, all the world is a nail, goes the adage. Even a nail-filled world may seem attractive for purveyors of fine hammers, they should still be asking to what purpose the nails are to be used.
To whit for example, KPMG’s quick scan of unstructured data to identify (say) credit card numbers. Sure, it may serve a purpose. But the rhetoric — “Complete coverage, get in control over unstructured data on premises and in the cloud.” implies that a single piece of (no doubt clever) pattern matching software can somehow solve a goodly element of your GDPR woes.
As I have written before, if you want to get there, don’t start from the place which looks at data and says “Is this bit OK? What about this bit?” A better starting point is the regulation, its rules around the kinds of data you can process and why, as documented by the Information Commissioner’s Office (ICO). The “lawful bases” offer a great deal of clarity, and start discussions from the right point.
Mapping an understanding of what you want to do with data, against what data you need, is not cause for concern. In the vast majority of cases, this is no different to what you would do when developing an information management strategy, undertaking a process modelling exercise, or otherwise understanding what you need to do business efficiently and effectively.
The thing GDPR rules out is use of personal data people didn’t want you to have, to fulfil purposes they didn’t want you to achieve. For example, use of ‘cold lists’ by direct marketing agencies may become more trouble than it is worth — both the agency, and the organization contracting them, become culpable. Equally, selling someone’s data against their will. That sort of thing.
But meanwhile, if you were thinking of harvesting maximum amounts of data about, well, anybody, because you were thinking you could be monetizing or otherwise leveraging it, or you were buying data from others and looking to use it to sell people things, goods or services, you should probably look for other ways to make money that are less, ahm, exploitative.
But if you have concerns about GDPR, and you are ‘just’ a traditional business doing traditional kinds of things, you have an opportunity to revisit your information management strategy, policies and so on. If these are out of date, chances are your business is running less efficiently than it could be so, how about spending to save, building in compliance in the process?
Across the board right now, you can get up to speed with what GDPR means for the kind of business you run, using the free helplines the regulators (such as the ICO) offer. If you are concerned, speak to a lawyer. And indeed, talk to vendors and consulting firms about how they are helping their customers, but be aware that their perspective will link to the solutions they offer.
Thank you to Criteo and Veritas, whose briefings and articles were very useful background when writing this article. As an online display advertising firm, Criteo is keenly aware of questions around personal vs pseudonymous data, as well as the legal bases for processing. Veritas offers solutions for analysis of unstructured data sources, and has GDPR modules and methodologies available.
The current, “sudden” plague of deepfake videos is just the latest in a series of “unexpected” events caused by “unplanned” use of technology. More will occur, and indeed are already happening: in a similar vein the computer-generated mash-up videos on YouTube that care more about eyeballs than child protection; the ongoing boom in cyber-trolling; bitcoin pimping and pumping. To be expected are misuse of augmented and virtual reality, 3D printing and robotics. Wait, 3D-printing of guns is so five years ago.
As I’ve written before, such bleak illustrations are the yang to innovation’s ying: trolling, for example, is the downside to the explosion of transparency illustrated by the ongoing, global wave of #MeToo revelations (“revelations” in its traditional, not salacious media sense). The present day is multi-dimensional and complex, and it is often difficult to separate positives from the negatives: so much so that we, and our legislative bodies, act like rabbits in headlights, doing little more than watch as the future unfolds before our eyes.
Or, we try to address the challenges using ill-equipped mechanisms — was it Einstein who said, “We can’t solve problems by using the same kind of thinking we used when we created them”? Nice words, but this is what we are doing, wholesale and globally: lawmakers are taking fifteen years to create laws such as GDPR which, while good as they go, are also, immediately insufficient; meanwhile the court of public opinion is both creating, and driven by, power-hungry vested interests; and service providers operate stable-door approaches to policy.
What’s the answer? To quote another adage, “If you want to get there, don’t start from here.” We need to start our governance processes from the perspective of the future, rather than the past, assessing where society will be in five, ten, fifteen years’ time. In practice this means accepting that we will be living in a fully digitized, augmented world. The genie is out of the bottle, so we need to move focus from dealing with the potential consequences of magic, and towards accepting a world with genies needs protections.
In practical terms, this means applying the same principles of societal fair play, collective conscience and individual freedom to the virtual world, as the physical. I’m not a lawmaker but I keep coming back to the idea that our data should be considered as ourselves: so for example, granting access to a pornographic virtual or 3D-printed robot representation of an individual, against their will, should be considered to be abuse. It’s also why speed cameras can be exploitative, if retrofitted to roads as money generators.
Right now, we are trying to contain the new wine of the digital age in very old, and highly permeable skins created over previous centuries. I remain optimistic: we shall no doubt look back on this era as a time of great change, with all its ups and downs. I also remain confident in the democratizing power of data, for all its current, quite messy state, and that we shall start seeing more tech-savvy approaches to legal and policy processes.
Meanwhile, perhaps we shall rely on younger, ‘digital native’ generations to deliver the new thinking required, or maybe — is this too big an ask? — those currently running our institutions and corporations will have the epiphanies required to start delivering on our legislative needs, societal or contractual. Yes, I remain optimistic and confident that we will get there; however, when this actually happens is anybody’s guess. We are not out of the woods yet.
Every company’s data is getting larger and more important to its survival and success. Companies everywhere are realizing that data is a key asset that can directly impact business goals. Data is projected to be the top item of spend in many industries. Most companies have to admit that no matter what business they are in, they are in the business of information.
Data is valuable. It gives a business the view it needs to understand and improve itself. Data is proving to be the next natural resource for the enterprise. “Let no data escape” must be the mantra of our systems development. We need to glean all possible value out of every data element and we need a modern data architecture to enable this strategy.
This mantra implies that we must grow the data science of our organization along with our data architecture to fully deal with the many forms of data.
All of this cannot be accomplished without an intense focus on the various and increasing number of technical bases that can be used to store, view, and manage data. There are more than ever now that have merit in organizations today.
Yet many organizations continue to push forward month after month with legacy technology that did a good job in the past and that continues to parlay the skillset of the organization. However, data technology and data science have progressed with the importance of data. It is imperative to raise the data foundation of your company to be able to cultivate it as an asset of the organization.
The benefits of modern data architecture are as follows:
It ensures the ability of the data analysis function of the organization to actually do analysis rather than restrict it to data hunting and preparation almost exclusively.
It provides the ability to maneuver as an organization in the modern era of information competition with consistent, connected data sets with every data set playing a mindful and appropriate role.
It enables a company to measure and improve the business with timely key performance indicators, such as streamlining your supply chain or opening up new markets with new products and services supported by technology built for analytics.
This paper will help an organization understand the value of modernizing its data architecture and how to frame a modernization effort that delivers analysis capabilities, diverse yet connected data, and key performance measures.
The majority of the information in your organization that is not under management is unstructured data. Unstructured data has always been a valuable asset to organizations, but it can be difficult to manage. Emails, documents, medical records, contracts, design specifications, legal agreements, advertisements, delivery instructions, and other text-based sources of information do not fit neatly into tabular relational databases. Even many NoSQL databases and Hadoop do not adequately address the specialized pre-processing, query, and organization requirements of pure unstructured data, and instead rely on techniques that essentially structure the data first.
There is no one set of tools that will solve everything. However, if you have a heavy unstructured data workload and wish to optimize your results, a new breed of powerful search and data management tools is changing the game. These tools, which are poised to expand dramatically within organizations that realize the gravity of the challenge, have been chosen for this report.
This Sector Roadmap is focused on unstructured data management tool selection for multiple uses across the enterprise. We eliminated any products that may have been well-positioned and viable for limited or non-analytical uses, such as log file management, but deficient in other areas. Our selected use cases are designed for high relevance for years to come and so the products we chose needed to match all these uses. In general, we recommend that an enterprise only pursue an unstructured data management tool capable of addressing a majority or all of that enterprises’ use cases.
Organizations today need to take advantage of the numerous relevant data platforms, while maintaining a central repository where governance can be enacted and quality can be assured. Managing the data effectively is a key indicator of success in analytics. Progressive organizations have more data platforms than ever before, and there is a clear need to bring key data together for the entire company. However, with hybrid and cloud architectures—key data from sources and for target systems distributed among on-premises, cloud, and third-party systems—the data management challenge is moving exponentially.
Success in data, analytics, and even their business is concomitant to an enterprise’s ability to manage and glean insight from unstructured data with a modern platform and mature process.
In this Sector Roadmap, vendor solutions are evaluated over five Disruption Vectors: query operations, search capabilities, deployment options, data management features, and schema requirements.
Key findings in our analysis include:
Number indicates company’s relative strength across all vectors
Size of ball indicates company’s relative strength along individual vector
Today’s enterprise data requirements are clearly dividing into a need for latency-sensitive and capacity-driven solutions, as organisations store and exploit data from existing and increasingly machine generated sources. This webinar looks at how enterprises meet the demand for capacity-driven data with object storage solutions from the major and upcoming solution vendors.
Join Gigaom Research and Hitachi Data Systems (HDS) for this free expert webinar.
During the webinar you will learn:
Factors driving the adoption of object storage
Critical features to look out for in object storage solutions
Analysis of vendor offerings available in the market today
Gigaom’s assessment of the market leaders and followers
Who Should Attend:
CxOs; including COO, CIO, CTO
Head of IT, Storage, Infrastructure
VP, Architect, Mobility, BYOD
Cloud and Software Architects
App Owners, Leads in App Dev
Chief Security / Compliance Offer
General Counsels, Records Manager
Line-of-Business Leaders: Sales, Customer Service, Marketing, Finance, HR or New Business Development, Product Managers
About Chris Evans, Gigaom Analyst
Chris Evans is a Gigaom Analyst and IT consultant with over 28 years of experience. Over his career, Chris has provided consultancy and advice to a wide range of customers and industry segments, including finance, utilities and IT organizations. His expertise is focused on resolving IT related business issues.
Chicago ITW 2017: Sagi Brody, Webair’s CTO is hoping that the recent wannacry ransomware attack brings to light the importance of disaster recovery. After all, disaster recovery is in Webair’s domain. The company has created a Disaster Recovery as a Service (DRaaS) offering aimed at helping businesses instantly recover from most any IT related problem. “The difference in our offering is that we allow our customers to recover individual applications, instead of forcing them into an all or nothing approach” said Brody. For today’s enterprises, that is a critical difference, especially since disasters come in all shapes and sizes.
Take for example last week’s wannacry attack, which rapidly spread and held companies for ransom, at least as far as access to their data was concerned. As a windows centric attack, not all infected sites had all applications impacted. “With an all or nothing approach, those impacted would have to roll back all operations to a point before the attack” added Brody. Brody said “with an application centric approach, businesses could focus on restoring line of business apps, while also remediating systems, creating the quickest path to productivity.”
While speed to recovery is an important aspect of recovering from a disaster, there is also another critical element, one that can be summed up as “at what point did my data go bad?”
For some disaster recovery solutions, backups only occur daily, happening overnight, meaning that when there is some type of failure, the data is only as good as the last backup – potentially hours of productivity could be lost, especially in a transaction sensitive environment.
Very few businesses could afford to lose a day’s worth of sales transactions, or other activity, without productivity being impacted. Brody said “Our technology closes that backup window to something that can be thought of in seconds. With wannacry, some businesses wound up losing hours or days of work, simply because that could not restore data from right before the attack manifested.”
Brody added “our solution makes it possible to recover applications right up to when a problem has manifested, potentially saving hundreds of hours of work.”
For many enterprises, the idea of having application awareness as part of a DR solution should be a critical consideration, especially as the potential for disruption increases and unexpected problems begin to manifest. Simply put, it all comes down to management, the ability manage operations effectively when they work, and more importantly when they don’t.
Chicago, ITW 2017: Undersea connectivity provider Aqua Comms is looking to shake up the world of cross Atlantic connectivity by introducing subsea fibre optic networks that traverse different routes. Based in Ireland, Aqua Comms came into existence to meet an underserved market, one that is experiencing phenomenal growth. Aqua Comms CEO Nigel Bayliff told GigaOM “We are seeing that the fastest growing data market is the one supporting Europe to North America connectivity.”
With that in mind, Aqua Comms set out to meet that exploding need for communications and deployed three new undersea cables, but with an added twist, the company built enhanced security into the equation by routing cables away from shipping and fishing areas to avoid accidental cable damage. What’s more, the company has also focused on having the most secure landfalls to protect connectivity even further. While security and reliability can be credited for helping AquaComms grow, there are multiple other reasons for connectivity growth.
Bayliff attributes many factors to the company’s growth, as well as the increase in traffic being carried. “More and more enterprises are looking for more efficient ways to connect and enhance connectivity between north America and Europe” Bayliff said.
Bayliff added “not so surprisingly, much of the demand is being generated by the growth of the cloud, as well as enterprises moving towards digital transformation. Both of which are demanding low latency and increased bandwidth.”
However, it is not just typical or expected business needs that are driving the need for more connectivity and increased capacities. Bayliff said, “Data center connectivity has become a big driving factor, thanks to the rapid growth of data centers in Ireland, which in its own right is becoming a major player in the data center space.”
While numerous companies are looking at the advantages offered by Irish data centers, another issue comes to mind. “Content providers such as Facebook, Microsoft, Google and Amazon are taking an active role in subsea networks, and those companies have become major capacity buyers of various submarine systems.” Said Bayliff
One thing is certain, growing data needs will drive connectivity and companies such as Aqua Comms must rise to the surface to offer undersea connectivity that matters.
What are the biggest issues facing Big Data and marketing in 2017?
There are three primary issues currently facing “Big Data” and Marketing. First, the issue of irrelevance regarding the vast majority of collected data – and conversely, the power of focused, actionable insights from the right slice of data. The pressure on today’s data and analytics professionals continues to grow, forcing them to sift through the noise to find the hidden diamonds.
Second, consumers are changing their behavior faster than data collection and algorithms teams can handle. This leads to marketers chasing trends and opportunities, as they are overwhelmed with data from older paradigms. For example, marketers with a 360-degree view of the customer built in the 1990s, rebuilt in the 2000s and updated again after 2015, completely miss the power of social media while reacting to real-time events.
Third, and most importantly, while the behavioral data shared from our devices gives us some information, the intuition and insights from data science, operations and customer service teams remain equally, if not more so, important. Each business, whether an auto manufacturer or a local pizza pub, needs both the data and the on-the- ground insight to find the golden nuggets and avoid potential disasters.
What new technologies should marketers be using to better analyze and act on data?
It’s no longer a question of if marketers are adopting technology; it’s a matter of what kind of technology. Technology is critical for connecting data silos and catching shoppers’ attention. Brands should tap solutions that help them better understand consumer behavior across print, digital (mobile, video, etc.), in-store, TV and e-commerce channels to get the full scope of their audiences’ actions.
We are especially interested in technology that helps us visualize trends and detect changes in consumer behavior quickly. Consumers tend to lead brands into new arenas by adopting technologies, apps and devices faster than clients can adapt. Marketers have access to this comprehensive data intelligence and ultimately need to adjust their media strategies to take advantage of new behavior patterns, dynamically changing consumer segments and shopper preferences. Segmentation tools, technology that accounts for both integrated online and offline targeting and utilizing vendors with open partner ecosystems should all be part of a marketer’s arsenal. By not accounting for all media channels, marketers are missing the bigger picture. They are formulating a customer profile based on a partial view – technology must be utilized to gain a more holistic audience overview.
It’s important to note that while technology can clearly improve marketing results, it must be combined with a human touch to ensure it’s a more nuanced experience.
How can marketers tread the line between personalization and targeting while ensuring consumer privacy is respected?
It’s no surprise that privacy is a major issue today – consumers increasingly feel “creeped-out” as marketers aim to provide a relevant experience. We especially see this across mobile platforms with location data. Many consumers actually don’t mind sharing this data – it’s when brands use the information in unintended manners or without providing a relevant ROI for customers, so to speak, that brand/consumer friction appears. To avoid this, marketers need to build trust with their audience. This can be achieved by ensuring that the collected data benefits the consumer.
Another way to build trust is to establish a clear opt-in option. Marketers shouldn’t be luring their audience into opting-in – it should be a transparent process laying out exactly what the consumer can expect to receive in return. What if a consumer says “no?” Leave it be. Don’t annoy them with constant opt-in requests. If and when they’d like to share data, you can bet a new app or value proposition will draw them in.
What are some strategies that brands/marketers can leverage to make data as valuable as possible for consumers?
As shoppers, we all want to feel valued and understood. Knowing this mindset, it’s up to marketers to engage their audience in meaningful ways across the channels they prefer. Brands should use the data at their disposal for a catered experience. If a shopper has a birthday coming up, send a special discount code. Want to take it a step further? Allow them to share the code with up to five friends. This ends up being a win-win for the consumer and brand. The shopper feels valued, and the brand extends its reach to five potential new customers.
Data should also be tapped to include real-time offers in marketing campaigns. Imagine a consumer is departing work for the day and looking for a local spot to pick up some pizza for dinner. A simple mobile alert offering a pizza coupon in that exact neighborhood can be the needed influence to spur a shopper to make a purchase decision. Making data relevant for consumers shouldn’t feel like a hassle for marketers. The little extra effort can go a long way in customer engagement, retention and happiness. Yes – happiness.
Fluent in applying data science to business applications and driving revenue, Greg possesses nearly 25 years of industry experience. In his role as Chief Data and Analytics Officer at Valassis, he brings a unique combination of expertise across disciplines including data management, consumer analytics, marketing tech, sales operations and product development.
Read Full Article
Read for later
Articles marked as Favorite are saved for later viewing.
Scroll to Top
Separate tags by commas
To access this feature, please upgrade your account.