2018-07-24
大数据已经过了炒作阶段，但企业是否真的经受住了挑战并从大数据技术中获得了最大的价值？不见得。以下是大数据成熟度的现状和仍然存在的挑战。
围绕“大数据”这个词汇的炒作已经减少，但这并不意味着企业已经停止了将更多数据纳入其分析实践的工作，他们有时会使用Hadoop和Spark等技术。与其他许多已经过了高峰炒作期的技术一样，大数据技术已经走过了很长的一段路。尽管不是主流，但它们比几年前更接近企业生产。（注：Hadoop是一个开源的可运行于大规模集群上的分布式并行编程框架，其最核心的设计包括MapReduce和HDFS。基于 Hadoop，你可以轻松地编写可处理海量数据的分布式并行程序，并将其运行于由成百上千个结点组成的大规模计算机集群上。Spark是一个围绕速度、易用性和复杂分析构建的大数据处理框架。Spark提供了一个全面、统一的框架用于管理各种有着不同性质[文本数据、图表数据等[的数据集和数据源[批量数据或实时的流数据]的大数据处理的需求。Spark可以将Hadoop集群中的应用在内存中的运行速度提升100倍，甚至能够将应用在磁盘上的运行速度提升10倍。）
问题是“为什么”。26％的首席信息官（CIO）表示，企业智能和分析将帮助他们将自己的业务与竞争对手区分开来，将其作为最重要的投资优先事项。但是，91％的企业尚未达到数据和分析的“转型”成熟度水平。达到该水平的数据和分析是企业的核心基础，这一点非常重要，因此董事会设立了首席数据官。
那么断节出现在哪里？企业智能转向Hadoop/大数据关联公司的过程中。AtScale的大数据发展在各种规模的企业中位居前列。自2016年以来，该公司定期发布对企业大数据成熟度水平（企业大数据的状态）的研究报告。AtScale对全球超过429家公司的5593名数据专家进行了调查，受访者来自其客户群和调查合作伙伴，包括所有三家Hadoop分销商、Tableau、以及Linux和Apache基金会。考虑到受访群体，调查信息更接近已经在使用Hadoop等大数据技术或者倾向于这样做的企业。2018年的报告概述了这些企业面临的主要挑战、机遇和关注点。
一个更有趣的发现是，企业对他们在2018年的努力有点过分自信。今年，78％的受访者将其大数据成熟度评为“中”或“高”。但根据AtScale对这些企业的评级方法，只有12％的企业达到了“高”成熟度。
这些企业面临的最大挑战之一是几十年来一直存在于企业中的一种挑战：孤立的数据和分析方法。55％的受访者仍然使用孤立分散的分析方法。AtScale报告称，在线和公用事业行业在这一领域处于领先地位，并建立了卓越中心。金融服务和电信行业则比较滞后。
在未来数月和数年内，云可能会在数据和分析方面发挥更大的作用。77％的受访者表示他们将使用云处理大数据。此外，11％的受访者表示他们计划将Google BigQuery投入生产，60％表示正在调查BigQuery。超过40％的受访者表示他们会考虑使用云而不是本地解决方案。
虽然大数据可能会在很多情况下转向云，但这并不意味着此举只有好处。AtScale的调查发现，59％的受访者在云中部署了大数据，而去年的比例为53％。但此举干扰了最终用户访问数据的能力。自助服务访问率从去年的47％降至42％。
过去几年中，微软Power BI获得了很大的发展，AtScale的调查显示了这一点。受访者被要求为大数据选择首选的企业智能工具，前三名分别是Tableau、Microsoft Excel和Power BI。但是AtScale称，Power BI在去年的排名是第7位，这是一个很大的提升。
企业中的工具和平台不断增加，而分散、孤立的数据和分析工作成为受访数据专家关心的问题。数据治理从2016年的第五位上升至2018年的第二位。技能连续三年排在首位，是最严峻的挑战。
AtScale首席执行官戴夫�马里亚尼（Dave Mariani）在接受InformationWeek采访时说：“本地部署Hadoop是很难的。除了少数几家拥有技能的企业之外，其他企业很难管理Hadoop。还有一些企业完全跳过本地部署。”
随着云中部署大数据分析的普及，明年的大问题可能会完全转移到另一个领域。根据AtScale首席营销官布鲁诺�阿兹扎（Bruno Aziza）的说法，企业可能会担心未来云锁定的风险——与一家云供应商合作，当情况发生变化或者他们发现另一家供应商可以提供更好的服务时，却很难迁移。这些企业正在寻求多云策略，但随着时间的推移，他们可能会发现自己使用一家供应商的更多工具，因为它们能够更好地与平台保持一致。
4/2/2018
08:00 AM
Big data is past the hype stage, but are organizations really past the challenges and getting the most value out of big data technologies? Not so much. Here's a look at the state of big data maturity and the challenges that remain.
The hype around the term "big data" has certainly fallen away, but that doesn't mean that organizations have stopped their work to incorporate volumes more data into their analytics practices, sometimes using technologies such as Hadoop and Spark. As with many other technologies that have passed their peak hype, big data technologies have come a long way since then. They are closer to enterprise production than they were just a couple years ago, although they certainly aren't mainstream yet.
The question is why. A full 26% of CIOs said that business intelligence and analytics would help them differentiate their businesses from their competitors, making it the top investment priority. But 91% of organizations had not yet reached a "transformational" level of maturity in data and analytics. That level is really where data and analytics are a central underpinning of the business, so important that the chief data officer will sit on the board of directors.
So where is the disconnect? Business intelligence to Hadoop/big data connection company AtScale has had a front row seat to the evolution of big data in organizations both big and small. Since 2016 the company hasreleased regular research on organizations' big data maturity levels -- sort of the state of big data in organizations. AtScale has surveyed more than 5,593 data professionals at more than 429 companies globally, pulling from its own customer base and that of its survey partners, including all three Hadoop distribution vendors, plus Tableau, and the Linux and Apache Foundations. Given the base, the survey information is closer to representing organizations that are already using big data technologies such as Hadoop, or that are more inclined to do so. The 2018 report provides a snapshot of those organizations' top challenges, opportunities, and concerns are today.
Among the more interesting findings is that organizations are a bit overconfident about their efforts in 2018. This year, 78% of respondents ranked their big data maturity as medium or high. But according to AtScale's methodology rating those same organizations, only 12% have a high level of maturity.
One of the top challenges these organizations are facing is one that's been present in enterprises for decades -- a siloed approach to data and analytics in the organization. A full 55% of respondents are still dealing with siloed, decentralized analytics. AtScale reports that online and utilities verticals are leading in this area, having established centers of excellence. Financial services and telecommunications verticals are lagging.
The cloud may play a bigger role in data and analytics in the months and years ahead. A full 77% of respondents said they would use the cloud for big data. Further, 11% said they are planning to put Google BigQuery into production, and 60% are investigating BigQuery. More than 40% of respondents said they would consider the cloud instead of an on-premises solution.
While big data may be headed for the cloud in many cases, that doesn't mean the benefits of that move are universal. AtScale's survey found that 59% of respondents had deployed big data in the cloud, up from 53% last year. But the move had disrupted their end-users' ability to access the data. Self-service access fell to 42% of organizations, down from 47% last year.
Microsoft Power BI has gained a lot of ground in the past few years, and AtScale's survey shows just how much. Survey respondents were asked to name their top BI tool of choice for big data, and the top three were Tableau, Microsoft Excel, and Power BI. But that was a big jump in the rankings for Power BI, which had been in 7th place last year, AtScale said.
Tools and platforms are proliferating in the enterprise, and decentralized, siloed data and analytics efforts are a concern for data professionals surveyed. Data governance ranked as the number two concern in 2018, up from the fifth position in 2016. Skill sets have remained in the number one position as the top challenge for the three years the survey has been conducted.
"On-premises Hadoop is hard," said AtScale CEO Dave Mariani, in an interview with InformationWeek. "It's hard to manage for all but a few enterprises that have the skill set. There are a number of companies that are skipping on-premises entirely."
With the popularity of cloud for deployment of big data analytics, next year's big concern may shift to another area entirely. Organizations may be concerned about the risk of cloud lock-in going forward -- aligning themselves with one cloud vendor and then finding it difficult to move to a different one if circumstances change or they find a provider that offers better alignment, according to Bruno Aziza, CMO at AtScale. These organizations are seeking a multi-cloud strategy, but as time goes on they may find themselves using more of one vendor's tools because they are better aligned with the platform.
附件：
《Big Data Maturity - You're Not As Mature As You Think》--原文.pdf
《Big Data Maturity - You're Not As Mature As You Think》--译文.pdf
微信公众号