Why every TensorFlow developer should know about TFRecord!
Why waste time on maintaining your datasets and its respective labels on different files and why read it at different times, when you can do it in one place! This native file format used in Tensorflow allows you to shuffle, batch and split datasets with its own functions. Cool right?!
Subscribe to our awesome Newsletter.
After few days of Tensorflow, every beginner will meet this crazy awesome Tensorflow’s file format called Tfrecords. Most of the batch operations aren’t done directly from images, rather they are converted into a single tfrecord file (images which are numpy arrays and labels which are a list of strings). It’s always been a Beginner’s Nightmare to understand the purpose of this conversion and the real benefit this has to the workflow. So here I am making it easier to understand with simple to complex examples.
WHAT IS TFRECORD?
As per Tensorflow’s documentation,
“… approach is to convert whatever data you have into a supported format. This approach makes it easier to mix and match data sets and network architectures. The recommended format for TensorFlow is a TFRecords file containing tf.train.Example protocol buffers (which contain Features as a field).“
So, I suggest that the easier way to maintain a scalable architecture and a standard input format is to convert it into a tfrecord file.
Let me explain in terms of beginner’s language,
So when you are working with an image dataset, what is the first thing you do? Split into Train, Test, Validate sets, right? Also we will shuffle it to not have any biased data distribution if there are biased parameters like date.
Isn’t it tedious job to do the folder structure and then maintain the shuffle?
What if everything is in a single file and we can use that file to dynamically shuffle at random places and also change the ratio of train:test:validate from the whole dataset. Sounds like half the workload is removed right? A beginner’s nightmare of maintaining the different splits is now no more. This can be achieved by tfrecords.
Let’s see the difference between the code -
Naive vs Tfrecord
Follow the five steps and you are done with a single tfrecord file that holds all your data for proceeding.
tf.python_io.TFRecordWriterto open the tfrecord file and start writing.
- Before writing into tfrecord file, the image data and label data should be converted into proper datatype. (byte, int, float)
- Now the datatypes are converted into
- Finally create an
Example Protocol Bufferusing tf.Example and use the converted features into it. Serialize the Example using serialize() function.
- Write the serialized
If you closely see the process involved, it’s very simple.
Data -> FeatureSet -> Example -> Serialized Example -> tfRecord.
So to read it back, the process is reversed.
tfRecord -> SerializedExample -> Example -> FeatureSet -> Data
Reading from tfrecord
You can also shuffle the files using the