If you are a Data Engineer or Data Scientist who uses the Azure Cloud to build your analytics pipeline or service, you would frequently come across the problem of controlling who gets access to the resources in your pipeline and what they can do once they have access, in the case of a production deployment.

The Data Protection mechanisms in Azure are rooted in the main ideas listed below:

Identity: User or Groups who will access the service

Authentication: Verification of user or application as to proof of identity

Authorization: Privileges or access levels which allow the user or application…

The “worth” or value of networks is often expounded upon across a wide variety of subjects from the world wide web to the value of digitally-related companies. The bone of contention that arises mainly is, how can this be quantified with a mathematical base. There are a few empirical laws which try to do so and one the earliest such to emerge was from Robert Metcalfe, the inventor of Ethernet in the 1980s. Metcalfe’s law attempts to quantify the value of any network as being proportional to square of connected users of a system.

The foundation of Metcalfe’s law is…

Bloom Filters are quite simply put space-efficient, probabilistic data structures which help answer the question of set membership. From a mathematical lens say that you need to verify whether an element x belongs to a set A , if the Bloom Filter as a data structure returns true for x, what this means is that either x is definitely in the set A or it may not be in the set A. …

At it’s heart Azure Synapse Analytics, quite simply put is a Massively Parallel Processing (MPP) Service, which run on the Microsoft SQL Server engine, providing concurrency, elasticity and scalability. How Azure Synapse Analytics differs from other on-premises data warehouses lies purely in the manner that it has been architected for the Azure Cloud. When you consider a vanilla SQL server implementation, it runs on a Symmetric Multi-Processing (SMP) environment where a single storage point is coupled to a set of CPUs and queries which are run are then parallelized across the CPUs utilizing a service bus. Since all the CPUs…

When working with data engineering solutions in the real world, the main problem faced by data engineers in general is that data does not usually end up nicely formatted for analytics on it’s own ‘magically’ . It usually takes several preprocessing steps which involve getting data from the source, ingesting the data, doing processing in the forms of transformation on the data, persisting or storing the data and then the presentation or delivery of the data to the end consumer. …

To those whom learn Statistics in their high schools and undergraduate courses it comes across often as a rather “boring” , “pedantic”, “math-heavy” subject with no intrinsic real world applications. I feel that this maybe due to the fact that sometimes the motivation for learning the subject is not clearly presented to the students and this in turns leads to demotivation in terms of not being able to visualize why they (the students) would need it in real life. One of the most important applications of Statistics is how to make better decisions based on the underlying body of evidence…

As observed by scientists through the centuries, probability (and statistics) has been seen to underpin a lot of natural phenomena be it the distribution of heights of students in a classroom, to Born’s Rule for quantum systems (this was the reason for which Albert Einstein complained about God playing dice with the universe). Today probability and statistics taught at classrooms and undergraduate levels, ease the beginner to intuitively ‘grasp’ the concept of probability by usually introducing it as a mathematical concept which follows certain axioms , laying the foundations for more concepts such as random variables and probability distributions.

When…

This article has been in my drafts and I did not have time to work on it for a long time. I saw it was part complete and decided to finish and cap it off.

This is the follow up to my earlier article which touched briefly on Relational Databases and how in-memory technology is currently being leveraged with respect to Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP) workloads. …