Evolving ways of Data File Format

 

Data File Format define standard ways of storing information in a file or database. We require different file formats for different use cases.

For example — If we know that only Python systems are going to read our file then we can choose Pickle format as it is highly optimized.

CSV data format has been the most widely used option for data storage. Using CSV, we can read from and write to most data software. However, there is no schema attached and no standard way to control characters and it is not the best way to deal with complex data.

In this article, we will discuss the evolving ways of the data file format:

Parquet: Parquet is one of the most common data storage formats for Big Data as it is very fast. It also understands all the data types used by Pandas, including multi-index data frames. It is optimized to work with complex data in bulk and features different ways for efficient data compression and encoding types. It is mostly used as a data warehouse or data lake storage format.

Parquet is column-oriented unlike other data formats such as CSV which are row-oriented.The data for divided into column chunks and is written in the form of pages. Each page contains values for a particular column only, hence pages are best suited for compression as they contain similar values.

For example, if we have a Data Table in the form:

Data Table
Row-oriented storage format
Column-oriented storage format

The only issue with using Parquet is that it is not human readable.

Pickle: The pickle data format uses a relatively compact binary representation. This is a highly optimized method for data storage supported for python read and write. The only disadvantage of using Pickle is that it is only understandable by Python.

Thus, we can use Pickle only when the python system will read our files.

Feather: Feather is a fast and lightweight data storage format. It naturally understands almost all data types used by Pandas.

However, it can only be read and written from Python and a handful of other programming languages. It supports only non-nested and categorical (dictionary-encoded) data types.

A simple Hands-on exercise to try out the different file formats:

A sample data with variables and their data types

Refer to Github for code (Data_File_Formats.ipynb) and data (train.csv) file.

We converted the sample CSV data into different formats — pickle, feather and parquet. Then we compared the file size and read time taken.

Comparison of file size and read time taken

- The data file in CSV format is 5.8 MB in size. With Pickle and Feather, we can see a reduction in size up to 1.5MB. However, with parquet, we see a significant reduction in file size — 0.13MB

- Talking about the read time, while CSV file takes 111ms, Pickle and Feather have only 25 and 17ms respectively.

Factors to consider when choosing the data file format are:

Structure: A data file structure is a combination of representations for data in files and of operations for accessing the data. While CSV allows storage of nested data, Feather is only suitable for non-nested data.

Compatibility: CSV is compatible with most of the tools however Pickle is only supported by Python. Thus we cannot use Pickle if our files would be read by systems other than Python.

Schema: CSV does not have any schema attached while feather defines its own simplified schemas and metadata for on-disk representation.

Readability: It is easy to read a CSV file but Pickle is not human-readable.

Hey, if you like my content on Medium or Quantifiers and find it resourceful, you can show your support by hitting the clap button.

For PM Interviews, you can refer to amazing articles at Technomanagers


Comments

Post a Comment

Popular posts from this blog

Brain Teasers | Tiger and Sheep

Image
Question:   One hundred tigers and one sheep arc are put on a magic island that only has grass. Tigers can eat grass, but they would rather eat sheep Assume:    A. Each time only one tiger can eat one sheep, and that tiger itself will become a sheep after it cats the sheep.    B. All tigers are smart and perfectly rational and they want to survive. So will the sheep be eaten?      Solution :     Let no of tigers = n     Let's start with 1 tiger(n=1).   Since there is only one tiger it will definitely eat the sheep as there will be no more tigers to eat him once he becomes a sheep.       With n=2 tigers,   If any of the tigers eat the sheep, it will become a sheep itself so the other tiger who is left will definitely eat him. So none of the tigers will eat the sheep.     With n=3 tigers,   The sheep will be eaten since each tiger realizes that once it becomes a sheep there will be two tigers left and none of the two tigers will eat the sheep. So the first tiger to realize this will
Read more

Determine Weight of Counterfeit coins [Asked in GS Interview]

  Question :   There are 5 bags with 100 balls in each bag. A ball can weigh 9 grams, 10 grams or 11 grams. Each bag contains balls of equal weight, but we do not know what type of ball a bag contains. Cookie has a digital scale (the kind that tells the exact weight). How many times does she need to use the scale to determine which type of ball each bag contains?     Solution:   Step 1:   Cookie marks 9 grams ball as  - 1 , 10 grams as  0  and 11 grams as  +1  and bags as Bag1, Bag2.. Bag5       Now let us start with a simple version of this problem:   When no of bags = 1 i.e. we only have Bag1   We need to weigh only one of the balls from Bag1 to determine the type of ball present in   Bag1     When no of bags = 2   If  we pick 1 ball  marked 0  from Bag1  and another ball marked as 0 from Bag2 the sum is 0   Similarly, if Bag1 contains 1 ball marked -1 and Bag2 contains another ball marked as 1 then also sum is 0   So by taking only 1 ball from Bag1 and Bag2 each it will not be possi
Read more

Brain Teasers | Screwy Pirates

Image
  Question :   Source: Google Five pirates  looted a chest full of  100 gold coin s . Being a bunch of democratic pirates, they agree on the following method to divide the loot: The most senior pirate will propose a distribution of the coins. All pirates. including the most senior pirate, who will then vote. If at least 50% of the pirates (3 pirates in this case) accept the proposal, the gold is divided as proposed. If not, the most senior pirate will be fed to the shark and the process starts over with the next most senior pirate... The process is repeated until a plan is approved. You can assume that all pirates are perfectly rational: they want to stay alive first and to get as much gold as the possible second. Finally, being blood-thirsty pirates, they want to have fewer pirates on the boat if given a choice between otherwise equal outcomes. How will the gold coins be divided in the end?      Solution:     Since the solution for 1 pirate would be trivial let me start with 2 pirates
Read more