Data Engineer_L3

We are DATA Group, a global IT solutions provider. Our mission is to simplify our clients' lives with innovative technology.

We are looking for a Databricks Specialist Consultant to join our team in Lisbon, Portugal. As a key member of our data engineering team, you will be responsible for designing and implementing large-scale data architectures using Unity Catalog and Microsoft Purview.

**Job Summary:**

• Migrate Parquet files stored in Azure Storage Accounts to Delta Tables registered in Unity Catalog in Databricks.
• Apply granular access control at table, column, and schema level using RBAC in Unity Catalog, ensuring compliance and security.
• Configure and optimize Unity Catalog to provide centralized governance over all data in Databricks, ensuring that permissions and data lineage are clearly defined.

**Responsibilities:**

• Expertise in Unity Catalog:
  • Implement Bronze, Silver, Gold architecture in Databricks, with different layers of data for ingestion, transformation, and final exposure for reporting.
  • Create Databricks clusters suitable for each layer, optimizing performance and guaranteeing scalability and security in data processing.
• Advanced integration with Microsoft Purview:
  • Integrate Unity Catalog with Microsoft Purview for automatic cataloging, data lineage tracking, and auditing.
  • Ensure that all data changes, permissions, and metadata are visible and auditable via Purview, guaranteeing compliance with regulations such as GDPR and HIPAA.
• Notebook Conversion and Delta Table Optimization:
  • Review and migrate existing notebooks that handle Parquet files to use Delta Tables registered in Unity Catalog.
  • Implement performance optimizations in Delta Tables, using commands such as OPTIMIZE and VACUUM to improve query efficiency and free up space.

**Required Skills and Qualifications:**

• Expert in Unity Catalog:
  • Advanced experience in configuring, managing, and optimizing Unity Catalog in Databricks, including access control, security policies, and governance.
• Microsoft Purview:
  • Proficiency in Microsoft Purview, with experience in integrating and maintaining data governance with Unity Catalog, ensuring traceability, auditing, and compliance.
• Databricks and Delta Lake:
  • Solid experience using Databricks for large-scale data manipulation, especially utilizing Delta Lake and Delta Tables.
  • Ability to implement and optimize data pipelines in Bronze, Silver, and Gold tiers in Databricks.
• Azure Storage:
  • In-depth knowledge of Azure Storage Accounts, Azure Blob Storage, and Azure Data Lake Storage (ADLS), including the manipulation of Parquet files for storage and performance optimization.
• Power BI:
  • Ability to integrate Databricks data into Power BI to create dynamic dashboards and reports, ensuring security permissions are respected.

The salary for this position is around €85,000 per year, depending on experience.