• Latest
  • Trending
Stream Data Processing with Apache Kafka and TensorFlow

Stream Data Processing with Apache Kafka and TensorFlow

December 16, 2021
Inaugural AfCFTA Conference on Women and Youth in Trade

Inaugural AfCFTA Conference on Women and Youth in Trade

September 6, 2022
Instagram fined €405m over children’s data privacy

Instagram fined €405m over children’s data privacy

September 6, 2022
8 Most Common Causes of a Data Breach

5.7bn data entries found exposed on Chinese VPN

August 18, 2022
Fibre optic interconnection linking Cameroon and Congo now operational

Fibre optic interconnection linking Cameroon and Congo now operational

July 15, 2022
Ericsson and MTN Rwandacell Discuss their Long-Term Partnership

Ericsson and MTN Rwandacell Discuss their Long-Term Partnership

July 15, 2022
Airtel Africa Purchases $42M Worth of Additional Spectrum

Airtel Africa Purchases $42M Worth of Additional Spectrum

July 15, 2022
Huawei steps up drive for Kenyan talent

Huawei steps up drive for Kenyan talent

July 15, 2022
TSMC predicts Q3 revenue boost thanks to increased iPhone 13 demand

TSMC predicts Q3 revenue boost thanks to increased iPhone 13 demand

July 15, 2022
Facebook to allow up to five profiles tied to one account

Facebook to allow up to five profiles tied to one account

July 15, 2022
Top 10 apps built and managed in Ghana

Top 10 apps built and managed in Ghana

July 15, 2022
MTN Group to Host the 2nd Edition of the MoMo API Hackathon

MTN Group to Host the 2nd Edition of the MoMo API Hackathon

July 15, 2022
KIOXIA Introduce JEDEC XFM Removable Storage with PCIe/NVMe Spec

KIOXIA Introduce JEDEC XFM Removable Storage with PCIe/NVMe Spec

July 15, 2022
  • Consumer Watch
  • Kids Page
  • Directory
  • Events
  • Reviews
Saturday, 28 January, 2023
  • Login
itechnewsonline.com
  • Home
  • Tech
  • Africa Tech
  • InfoSEC
  • Data Science
  • Data Storage
  • Business
  • Opinion
Subscription
Advertise
No Result
View All Result
itechnewsonline.com
No Result
View All Result

Stream Data Processing with Apache Kafka and TensorFlow

by ITECHNEWS
December 16, 2021
in Data Science, Leading Stories
0 0
0
Stream Data Processing with Apache Kafka and TensorFlow

As one of the most popular deep learning frameworks, TensorFlow has been used widely adopted in production across a broad spectrum of industries. The upcoming TensorFlow 2.0, which was announced recently, will be released early this year with many changes. The high-level tf.keras API, eager execution, and tf.data greatly simplify the usage. Here is a good example of model training in several lines:

“`

import tensorflow as tf

mnist = tf.keras.datasets.mnist

 

(x_train, y_train),(x_test, y_test) = mnist.load_data()

x_train, x_test = x_train / 255.0, x_test / 255.0

 

model = tf.keras.models.Sequential([

 tf.keras.layers.Flatten(input_shape=(28, 28)),

 tf.keras.layers.Dense(512, activation=tf.nn.relu),

 tf.keras.layers.Dropout(0.2),

 tf.keras.layers.Dense(10, activation=tf.nn.softmax)

])

model.compile(optimizer=‘adam’,

             loss=‘sparse_categorical_crossentropy’,

             metrics=[‘accuracy’])

 

model.fit(x_train, y_train, epochs=5)

model.evaluate(x_test, y_test)

 

“`

YOU MAY ALSO LIKE

Inaugural AfCFTA Conference on Women and Youth in Trade

Instagram fined €405m over children’s data privacy

However, the data input processing component tf.data in TensorFlow only covers a small set of file formats. Users from different industries often encounter challenges integrating TensorFlow with data sources not commonly seen in the machine learning community.

One example is the integration of TensorFlow with Apache Kafka. Kafka is widely used for stream processing and is supported by most of the big data frameworks such as Spark and Flink. For a long time, though, there was no Kafka streaming support in TensorFlow. The data formats such as TFRecords and tf.Example in TensorFlow are also rarely seen in big data or data science community.

Many users are forced to consolidate these two frameworks in a very awkward way: setup another infrastructure, read messages from Kafka, convert the messages into TFRecord format, invoke TensorFlow to read the TFRecord from a file system, run the training or inference, and save the models or results back to the file system. This process is really error-prone and hard to maintain from an infrastructure perspective.

With help from the community, Apache Kafka streaming support for TensorFlow has been released recently as part of the tensorflow-io package (https://github.com/tensorflow/io) by TensorFlow’s SIG IO. SIG IO is a special interest group under TensorFlow organization, with a focus on I/O, streaming, and file formats support. In addition to Apache Kafka streaming, tensorflow-io also includes support for a very broad range of data formats and frameworks. It supports Apache Ignite for memory and caching, Apache Parquet and Arrow for serialization, AWS Kinesis and Google Cloud Pub/Sub for streaming, and many video, audio, and image file formats. Both Python and R language could be used, which are especially convenient to the data science community.

It is worth to mention that tensorflow-io is implemented as a part of the tf.data pipeline and natural extension of TensorFlow 2.0 API. In other words, users are able to read the data from Kafka and pass it to tf.keras. Training or inference is exactly the same simple and succinct way as the example shown at the beginning of this article.

By Yong Tang

ShareTweetShare
Plugin Install : Subscribe Push Notification need OneSignal plugin to be installed.

Search

No Result
View All Result

Recent News

Inaugural AfCFTA Conference on Women and Youth in Trade

Inaugural AfCFTA Conference on Women and Youth in Trade

September 6, 2022
Instagram fined €405m over children’s data privacy

Instagram fined €405m over children’s data privacy

September 6, 2022
8 Most Common Causes of a Data Breach

5.7bn data entries found exposed on Chinese VPN

August 18, 2022

About What We Do

itechnewsonline.com

We bring you the best Premium Tech News.

Recent News With Image

Inaugural AfCFTA Conference on Women and Youth in Trade

Inaugural AfCFTA Conference on Women and Youth in Trade

September 6, 2022
Instagram fined €405m over children’s data privacy

Instagram fined €405m over children’s data privacy

September 6, 2022

Recent News

  • Inaugural AfCFTA Conference on Women and Youth in Trade September 6, 2022
  • Instagram fined €405m over children’s data privacy September 6, 2022
  • 5.7bn data entries found exposed on Chinese VPN August 18, 2022
  • Fibre optic interconnection linking Cameroon and Congo now operational July 15, 2022
  • Home
  • InfoSec
  • Opinion
  • Africa Tech
  • Data Storage

© 2021-2022 iTechNewsOnline.Com - Powered by BackUPDataSystems

No Result
View All Result
  • Home
  • Tech
  • Africa Tech
  • InfoSEC
  • Data Science
  • Data Storage
  • Business
  • Opinion

© 2021-2022 iTechNewsOnline.Com - Powered by BackUPDataSystems

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In
Go to mobile version