001. Introduction to Azure Data Engineering

001- Overview

History of SQL

SQL Introduction Notes

002- Install SQL Server 2025

003.CreateAndDropDatabase

SQL Server Hierarchy

CreateAndDropDatabase Notes

004.SystemDatabases

SystemDatabases Notes

005.CreateAndDropTable

CreateAndDropTable Notes

006.InsertDataInTable

007.Select

SQL Quiz 5 – Select

008.InstallAdventureWorksSampleDB

009.GetUniqueValues

SQL Quiz 6 – Distinct

010.SortData

SQL Quiz 7 – Sort Data

011.Comments

012.FilterData

SQL Quiz 8 -Filter

013.FilterWithWildCharacters

SQL Quiz 9 – Filter

SQL Mega Quiz 1

014.AggregateFunctions

015.GroupingRows

016.Select..Into

017.CreateTableWithPrimaryKey

018.CreateTableWithNOTNULLConstraint

019.CreateTableWithUniqueConstraint

020.CreateTableWithCheckConstraint

021.CreateTableWithDefaultConstraint

022.CreateTableWithAutoIncrement

023.UpdateRowsInTable

024.DeleteRowsFromTable

025.StringFunctions-Part1

026.StringFunctions-Part2

027.Scenario-CombineNames

028-Scenario-ExtractFirstAndLastNames

029.DateTimeFunctions-Part1

030.DateTimeFunctions-Part2

031.DateTimeFunctions-Part3

032.DataType-Part1-Integers

033.DataTypes-Part2-ApproximateNumeric

034.DataTypes-Part3-DateTime

035.DataTypes-Part4-Strings

036.DataTypes-Part5-Uniqueidentifier

037.DataTypes-Part6-bit

038.DataType Conversion

039.Joins Part 1 – Introduction to joins

040.Joins Part2- Using Joins

041.Joins Part3- Join 3 tables

042.Joins Part3- Using Expressions

043.Joins Part4- Outer Joins

044.Scenario-Joins Part 6 – Self Join

045.FactAndDimensions

046.IIF Function

047.CASE

048.IIF & CASE multiple conditions

049.Scenario-Custom sorting

50.UNION & UNION ALL-H

051.INTERSECT

052.EXCEPT

053. Foreign key constraint

054.Subquery-PART 1

055.Subquery-PART2-IN & NOT IN

056.Subquery-PART3-Update

057.Subquery-PART4-Derived Table

058.Subquery-PART5-EXISTS, Correlated subquery

059.Scenario-Subquery-PART6-Get contribution yearwise

060. HAVING Clause

061.TOP Clause

062. Scenario- Get TOP & BOTTOM Products

063.Window Functions-PART1-ROW_NUMBER

064.Window Functions-PART2-RANK & DENSE_RANK

065.Window Functions-PART3-LAG & LEAD

066.Window Functions-PART4-FIRST_VALUE & LAST_VALUE

067. SQL Classification

068. ALTER TABLE -PART1- Columns

069. ALTER TABLE -PART2- Constraints

070. OFFSET

071.COALESCE Function

072.MERGE

073.SCHEMA

074. GROUP BY WITH ROLLUP

075. GROUP BY WITH CUBE

076.PIVOT

077.UNPIVOT

078.VIEWS

079.CommonTableExpression-Part1-Introduction

080.CommonTableExpression-PART2-MultiPart

081.CommonTableExpression-PART3-Recursion

082.Variables

083.IF..ELSE

084.WHILE LOOP

085.TempTables

086.StoredProcedures

087.Converting a table to JSON

088.CreatingNestedJSONOutput

089.outputClause

090.SlowlyChangingDimensions

091.Indexes-Part1(ClusteredIndex)

092.Indexes-Part2(NonClusteredIndex)

093.Indexes-Part3(Theory)

094.Indexes-Part4(IndexSeek)

095. Delete and Update on CASCASE

096. String_agg and String_split functions

097. Creating Functions

098. Cross apply and Outer apply

099. Quotename function

100. sp_executesql

101. SQL Injection

102. Dynamic Pivot

001.Find Average

002.JoinsChallenge

003.IdentifyAndDeleteDuplicates

004.CustomMerging

005.MissingData-FindMissingDepartments

006.Find2ndLargestSalary

007.2ndLargestSalaryInEachDepartment

008.RowNumbering

009.FindAlternateRows

010.Palindrome

011.FindDuplicateEmails

012.UpdateCorrectGender

013.TransformStudentInfo

014.DefectClassification

015.DailyRunningTotal

016.MTDTotal

017.QTDTotal

018.YTDTotal

019.PreviousYearSalesAndYOYGrowth

020.3 month moving sum

021.ArrangeRowsInAlternateWay

001.Introduction to DataWarehousing

002.DataLoads

003.StorageLayoutModels

004.RobinRoundDistribution

005.HashDistribution

006.ACID Properties

007.Normalization

001.CloudComputing

002.CloudProviders

003.AzureIntroduction

004.CreateAzureTrialAccount

005.GettingStartedWithAzure

006.CreateDataLake

007.CreateAzureSQLServerAndDatabase

008.CreateDataFactory

009.UpgradeToPay-as-you-go

Episode 1: The beginning

Episode 2: Getting started with data factory

Episode 3 : Activities

Episode 4 : Load Types

001.Introduction

002.CopyDataWithinDatalake

003.CopyEntireFolder_SpecificFiles

004.CopydatafromADLStoSQLDB_viceversa

005.CopyDataAdditionalColumns

006.SetVariableActivity

007.Copyfilesintimeframe

008.GetMetaDataActivity

009.ForEachActivity

010.CopyEachFiletoNewtable

011.Truncate&load

012.Copydatawithupsert

013.AppendVariableActivity

014.UsingSQLQueriesInCopyDataActivity

015.ColumnMappingInCopyDataActivity

016.DeleteActivity

017.CopyDataWithStoredProcedure

018.StoredProcedureActivity

019.LookupActivity

020.FilterActivity

021.IFActivity

022. SwitchActivity

023.ScriptActivity

024.ValidationActivity

025.ConvertCSVtoJSON

026.CopyJSONtoSQLDB

027.ExecutePipelineActivity

028.CopyDataIfFileExists

029.ParametersVsVariables

030.DeleteBlankFiles

031.CopyHeaderLessCSV

032. RetryLogic

033.CopyBehavior

034.MaxRowsPerFile

035.SplitDataSingleCriteria

036.SplitDataMultipleCriteria

037.ConsolidateDataFromMultipleFiles

038.ConsolidateDataFromMutipleFolders

039.CopyDataCustomMappings-SingleTable

040.CopyDataCustomMappings-MultipleTables

041.CopyDataPipeCharacter

042.CopyDataQuoteCharacter

043.IntroductionToDataFlows

044.SelectTransformation

045.SortTransformation

046.FilterTransformation

047.DerivedColumnTransformation

048.ConditionalSplitTransformation

049.CastTransformation

050.SurrogateKeyTransformation

051.AggregateTransformation

052.PivotTransformation

053.UnpivotTransformation

054.RankTransformation

055.WindowTransformation

056.UnionTransformation

057.LookupTransformation

058.JoinTransformation

059.ExistsTransformation

060.FlattenTransformation

061.ParseTransformation

062.StringifyTransformation

065.IntegrationRuntime

066.InstallSelfHostedIntegrationRuntime

067.CopyDataFromLocalToADLS

068.CopyDataFromLocaltoAzureSQLDB

069.CopyDataLocalSQLdbtoAzureSQLdb

070.CopyDataLocalsqlDBtoAzureSQLdb

071.UsingKeyVault

072.ScheduleTrigger

073.StorageEventTrigger

075.CopyFilesOnBasisOfFileTypes

076.IncrementalLoadFromADLStoSQLdbUsingStaging

077.IncrementalLoadFromADLStoSQLdbUsingDataflow

078.IncrementalLoadFromADLStoSQLdb-MultipleTablesUsingFileTimeStamp

079.IncrementalLoadFromADLStoSQLdb-UsingHashFunction

001.IntroductionToDatabricks

002. Signup For Databricks Free Edition

003.DatabricksPlatformOverview

001.IntroductionAndInstallation

Install VS Code

002.print

003.Variables

004.Concatenate

005.Interpolation

006.If..else

007.Input

008.IF..ELIF..ELSE

009.For..Loop

010.While..Loop

011.Break&Continue

012.Lists-Part1-IntroductionAndIndexing

013.Lists-Part2-Len,Existence&Loop

014.Lists-Part 3 – Add&RemoveItems

015.Lists-Part4-Count,Copy,Reverse&Sort

016.Lists-Part5-UpdatingLists

017.Set

018.Tuple

019.Dictionary

020.Comments

021.StringOperations-Part1

022.StringOperations-Part2

023.StringOperations-Part3

024.ListComprehensions

025.dir

026.DateTime-Part1-datetimeModule

027.DateTime-Part2-timedelta&timestamp

028.DateTime-Part3-monthdeltaModule

029.DateTime-Part4-relativedeltaModule

030.DateTime-Part5-Formatting

031.ExceptionHandling

032.None

033.Random

034.Functions-Part1-Introduction

035.Functions-Part2-ScopeOfVariable

036.Functions-Part3-Arguments

037.Functions-Part4-DocStrings

038.Lambda

039.Map

040.Reduce

041.Recursion

042.Generators

043.Decorators

001-GenerateAlphaNumericSeries

001. Flatten List

Quick Revision

002. Get Len Of Each Sublist In List

Quick Revision

003. Get Common Items in 2 Lists

Quick Revision

004.Sort Sublists In Order By Len

Quick Revision

005. Find missing numbers in list

Quick Revision

006. Zip and print elements in lists

007. Python Scenarios

001.IntroductionToBigData

002.EvolutionOfBigData

003.DistributedComputing

004.Features

005.HadoopEcosystem

006.TypesOfProcessing

001.IntroductionToDatabricksAndPyspark

002.IntroductionToApacheSpark

003.SparkComponentsAndAPI

004.SparkArchitecture

005.RDD

006.CreateRDDfromList

007.ControlPartitionsInRDD

008.CreateRDDfromTextfile

009.TransformationsOnRDD

010.LineageGraph

011.UnderstandingDAGfundamentals

012.MapreduceWorking

013.ReduceByKeyVsReduceByKeyLocally

014.GroupByKey

015.FilterTransformationOnRDD

016.SortBy&SortByKeyTransformationsOnRDD

017.ExtractTopBottomFromRDD

018.SaveRDDasTextfile

019.CoalesceAndRepartitionOnRDD

020.IntroductionToAccumulators

021.ImplementingAccumulators

022.BroadcastVariables

023.IntroductionToDataframes

024.ReadCSVInDataframe

025.DataFrameInsights

026.SelectColumnsFromDataFrame

027.AddModifyColumnsInDataFrame

028.DropColumnsInDataFrame

029.RenameColumnsInDataFrame

030.SortColumnsInDataFrame

031-FilterDataInDataFrame

032.RemoveDuplicatesFromDataFrame

033.CombineDataFrames

034.PatternBasedFiltersInDataframe

035.AddColumnOnBasisOfConditionInPyspark

036.CaseConversionInDataFrame

037.AggregationsOnDataFrame

038.AggregationsWithGroupBy

039.Pivot&UnpivotOnDataFrame

040.WindowFunctions

041.FillNullValuesInDataFrame

042.ReadCSVoptions

043.ImposeSchemaOnDataFrame

044.WriteModesInDataFrame

045.DateFunctions

046.ConvertRddtoDFandViceVersa

047.Explode

048.WorkingWithArraytypeColumn

049.dbutils

050.AccessADLSgen2UsingAccessKey

051.SignUpForAzureDatabricks

052.Widgets

053.CallingOtherNotebooks

054.ReadJsonFiles

055.ReadMultipleFilesAndGetFileNames

056.ParquetFileFormat

57.AccessADLSusingSAStoken

58.AccessADLSusingOAUTH

59.Read&WriteParquetFiles

060.DeltaFormat

061.CreateTempViews

062.CreateManagedAndUnManagedTables

063.Partitioning

064.Bucketing

065.ControlNumberOfRecordsWhileWriting

066.MaxPartitionBytes

067.SparkQueryExecutionPlans

068.Joins

069.SparkExecutionPlansForTransformations

070.BroadCastHashJoinAlgorithm

071.SortMergeJoinAlgorithm

072.ShuffleHashJoinAlgorithm

073.SortMergeBucketJoin

074.JoinHints

075.SparkSQL

076.SparkMemoryManagement

077.GarbageCollection

078.CPUTerminologies

079.DatabricksCompute&Clusters

080.ClusterManager

081.ResourceAllocation

082.DynamicAllocation

083.SerializationAndDeserialization

084.CacheAndPersist

085.HashFunctions

086.JDBC

087.IncrementalLoadADLStoSQLdb-SingleTableUsingWatermark

088.IncrementalLoadADLStoSQLdb-MultipleTableUsingWatermark

089.IncrementalLoadADLStoSQLdb-UsingHashFunction

090.IncrementalLoadADLStoSQLdb-UsingMetadata

InstallDocker

001.HistoryOfDataArchitectures

002.IntroductionToDeltaLake

003.ReadAndWriteDeltaFormat

004.UnderstandingDeltaLog

005.VersionHistoryAndTimeTravel

006.CheckPointing

007.CreateDeltaTable

008.GeneratedColumns

009.CreatePartitionedTable

010.SchemaEvolution

011.CopyInto

012.Merge

013.ColumnStatistics

014-Optimize-Vaccum-Zorder

015-LiquidClustering

016-ChangeDataFeed

017-ReorgTable

018.DeletionVectors

019.SCD-Type 1

020.SCD-Type 2

001.HiveMetaStore

002.IntroductionToUnityCatalog

003.SettingUpUnityCatalog

Managed Identity

004.CreateCatalog, Schema & Table

005.Autoloader

006.DataGovernanceOverview

007.ImplementingDataGovernanceUsingUnityCatalog

008.LakehouseFederation

009.LakehouseFederationForSQLServer

010.CopyFilesOnBasisOfFileTypes

011.VolumeInUnityCatalog

012.IntroductionToDeltaLiveTables(DLT)

013.DataQuality(Expectations)InDLT

001.Introduction

002.ReadStream

003.WriteStream

004.OutputModes

005.Sources&Sinks

006.Triggers

007.Joins

008.StatefulVsStateless

009.WindowOperations

001.IntroductionToAzureSynapseAnalytics

002.SQL Pools

003.DedicatedSQLPool

004.ServerlessSQLPool

005.DistributionTypes

006.ProvisionAzureSynapseAnalytics

007.WorkingWithServerlessSQLPool

001.IntroductionToAzureDevops

002.SignUpForAzureDevops

003.CreatingRepository

004.IntegrateDataFactoryWithAzureDevops

005.IntegrateDatabricksWithAzureDevops

006.SettingUpCICDForDataFactory

007.SettingUpCICDForDatabricks

008.LinkingKeyVaultToAzureDevops

ProjectLifeCycle

DevelopmentPhases

001.DSAIntroduction

002.Logarithm

003.ApproachTowardsProblemSolving

004.ComplexityAndBigONotation

005.LinearSearch

006.BinarySearch

001.IntroductionToJiraAndAgile

002.Scrum

003.JiraProjectTypes

004.JiraSignUp

005.CreatingScrumProject

006.CreatingBackLogVersionEpic

007.CreatingUserStories

008.CreatingSprint

009.CreatingSubtasks

001.Data Quality

002.UnderstandingRequirements

003.PurchaseDataModel

004.SalesModel

005.HRModel

006.ResourceSetUp

007.GettingStartedWithAzureDevops

008.UnderstandingDataSource

009.GettingDatabricksReady

010.DevelopingPurchaseModel-Part1

011.DevelopingPurchaseModel-Part2

012-TaskPlanning

013.DevelopingPurchaseBronzeNotebooks

Project Quiz 1

Project Quiz 2

Project Quiz 3

014.FirstDemo

015.DevelopingSilverNotebooks-Part1

016.DevelopingSilverAndOtherNotebooks

017.DevelopingOtherAndSilverNotebooks-Part2

018.CreatingRawAndBronzeSalesNotebooks

019.CreatingSalesSilverNotebooks-Part1

020.CreatingSalesSilverNotebookPart2

021.CreatingHrModelNotebooks

022.DevelopingDatabricksWorkflows

023.PowerBIDevelopmentKickOff

024. DevelopingPowerBiPurchaseReports-Part1

025.BugFixing

026.DevelopingPurchaseReport-Part2

IntroductionToDataQuality

ProjectFlow

SettingUpDataQualityInfrastructure

DevelopingRuleExecutionProcess

AutomatingBugTicketing

ImplementingNullCheck

001.IntroductionToFabric

002.SignupForFabricUsingPersonalEmail

003.CreatingAWorkspace

004.CreatingALakehouse

005.LoadDataIntoLakehouse

006.LoadDataFromFolder

007.SqlAnalyticsEndPoint

008.VisualQuery

009.DefaultSemanticModel

010.OneLakeExplorer

011.CreateShortcut

012.UpdateConnection

013.CreateTableUsingShortcut

014.CreatingAWarehouse

015.CreatingATableInWarehouse

016.CopyInto

017.LoadDataIntoWarehouseUsingPipeline

018.DataSharingBetweenLakehouseAndWarehouse

019.TableCloning

020.IntroductionToDataflow

021.IntroductionToNotebook

022.SparkOptions

023.Comments&SharingInNotebook

024.CodeSnippets

001.Introduction&Series

002.Describe&Agg

003.Indexing

004.Slicing

005.DataFrameBasics

006.Concat

007.Head&Tail

001.TrimAllColumns

002.CompareColumnsInDataframe

Power Bi Interview 1

Power Bi Interview 2 Level 1

Power Bi Interview 2 Level 2

Power Bi Interview 3

Data Engineering Interview 1

Introduction

001. Signup For Azure Databricks

002. Introduction To Apache Spark

003.Compute Types

Quick Revision

004. Spark Core Components

Quick Revision: Spark Core Components

005. Spark Architecture

006. Databricks Assistant

007. Create RDD from List

008. Understanding RDD Partitions

009. Managing RDD Partitions

Heads up : Unity Catalog

010. Hive Metastore

011. Unity Catalog

012. Setting up Unity Catalog

013. Introduction to Dataframes

Quick Revision

014. Create DataFrame from RDD

015. Create DataFrame from CSV

016. Best Practices to Read CSV

Common Dataframe Pyspark Transformations

017. Select Transformation

018.UnderstandingSparkExecutionPlans

019.DropColumns

020.Add Or Modify Column

021.Rename Columns

022. Sort Data

023. Filter Data

024. Drop Duplicates

025. Set operations

026. Add Column On Basis Of Condition

027. Aggregations

028. Pivot and Unpivot

029. Spark SQL

030. Date Functions

031. Window Functions

032. Dbutils

033. Widgets

034. Write Modes In Dataframe

035. Parquet File Format

036. Read and write parquet files

037. Understanding MaxPartitionBytes

038. Introduction To Delta Format

039. Managed and Unmanaged Tables

040. Partitioning

041. Bucketing

042. Query Files Using SQL

043. Joins

044. Join Algorithms

045. Optimizing SortMergeJoin

046. Spark Memory Management

047. Garbage Collection

048. CPU Terminologies

049. Resource Allocation

050. Dynamic Allocation

051. Serialization and Deserialization

052. Cache and Persist

053. Hash Functions

054. JDBC

055. Incremental Load ADLS to SQL db