Ilse Epskamp – Medium

Ilse Epskamp

SQL operations you might need one day

If you work with data you will most likely use SQL every now and then. If data is not populated in your SQL tables according to basic…

May 2, 2024

SQL operations you might need one day

May 2, 2024

Run integration test between Data Factory and Databricks in Azure DevOps CI pipeline

Check out this example of how to use Powershell to run an integration test between Azure Data Factory and Databricks in your CI pipeline.

May 2, 2024

Run integration test between Data Factory and Databricks in Azure DevOps CI pipeline

May 2, 2024

Published in
Azure Tutorials

Save Excel as CSV with custom delimiter

Let’s hope that you as a Data Engineer never need this article, because that means the data you are working with is by default…

Jun 8, 2023

Save Excel as CSV with custom delimiter

Jun 8, 2023

Building a scalable metadata-driven data ingestion framework

In this blog I discuss the concepts of a metadata-driven data ingestion framework, enabling scalable data processing and ingestion.

Oct 18, 2022

Building a scalable metadata-driven data ingestion framework

Oct 18, 2022

Published in
Azure Tutorials

CosmosDB (Gremlin) database design: positioning of edges in partitions

In this blog we will focus on CosmosDB data patterns, and will look specifically at the position and usage of edges in your graph database.

Jul 18, 2022

CosmosDB (Gremlin) database design: positioning of edges in partitions

Jul 18, 2022

Published in
Azure Tutorials

Add/Remove Azure Resource Locks with Powershell

Resource locks are a powerful mechanism to protect your resources from unauthorized operations.

May 30, 2022

Add/Remove Azure Resource Locks with Powershell

May 30, 2022

Published in
Azure Tutorials

Use dataset parameters to copy data to dynamically defined source and sink directories with ADF

In this blog we show how to configure dynamic source and sink directories for your Data Factory workflows.

May 2, 2022

Use dataset parameters to copy data to dynamically defined source and sink directories with ADF

May 2, 2022

Published in
Azure Tutorials

Do’s and Don’ts when working with CosmosDB Gremlin API

Sharing some of our experiences with ingesting data into Cosmos DB Gremlin API.

Apr 18, 2022

Do’s and Don’ts when working with CosmosDB Gremlin API

Apr 18, 2022

Published in
Azure Tutorials

Connect Azure Databricks to CosmosDB Gremlin API and run queries

How to connect Databricks to CosmosDB (Gremlin API) and run queries on the database including a scenario for usage in an end to end flow.

Apr 4, 2022

Connect Azure Databricks to CosmosDB Gremlin API and run queries

Apr 4, 2022

Published in
Azure Tutorials

Best practices when using Databricks notebooks in an automated workflow

Best practices or lessons learned when designing automated workflows with Databricks including a few examples of useful generic functions.

Mar 21, 2022

Best practices when using Databricks notebooks in an automated workflow

Mar 21, 2022

Published in
Azure Tutorials

Custom queue mechanism for Data Factory pipelines

How to queue your Data Factory pipeline so you run your pipeline sequentially.

Mar 7, 2022

Custom queue mechanism for Data Factory pipelines

Mar 7, 2022

Published in
Azure Tutorials

How to trigger Azure DevOps Release pipeline from Databricks

Learn how to use Python to trigger a DevOps Release pipeline using the Azure DevOps REST API.

Feb 21, 2022

How to trigger Azure DevOps Release pipeline from Databricks

Feb 21, 2022

Published in
Azure Tutorials

How to use the Data Factory Lookup activity to read data from storage during pipeline run

When designing a pattern with Data Factory, you can choose from a wide range of features and activities to handle your data requirements…

Feb 7, 2022

How to use the Data Factory Lookup activity to read data from storage during pipeline run

Feb 7, 2022

Published in
Azure Tutorials

Azure Storage: 5 tips for effective file system management

When handling large volumes of data you need to think about properly processing and storing the datafiles to ensure files are correctly…

Jan 24, 2022

Azure Storage: 5 tips for effective file system management

Jan 24, 2022

Published in
Azure Tutorials

How to pass parameters between Data Factory and Databricks

When working with data in Azure, running a Databricks notebook as part of a Data Factory pipeline is a common scenario. There could be…

Jan 10, 2022

Passing parameters between Data Factory and Databricks by Azure Tutorials

Jan 10, 2022

Ilse Epskamp

Ilse Epskamp

Azure Certified IT Engineer with 9+ years of experience in the banking industry. Focus areas: Azure, Data Engineering, DevOps, CI/CD, Automation, Python

Following

Help
Status
About
Careers
Press
Blog
Privacy
Rules
Terms
Text to speech