Job Description

Role: AWS Tech Lead with Databricks

Duration: Long Term

Location: Raritan, NJ- fully onsite

Overview:

An AWS Tech Lead with Databricks SME (Subject Matter Expert) knowledge is responsible for designing, deploying, and managing cloud-based infrastructure on AWS and supporting data solutions using Databricks. The role involves leading a team to ensure scalable, secure, and cost-efficient cloud services while integrating data pipelines and analytics solutions.

Roles and Responsibilities:

Cloud Infrastructure Management:

- Design, implement, and maintain AWS cloud solutions, including VPCs, EC2 instances, S3 storage, RDS, and networking configurations.

- Ensure high availability, disaster recovery, and backup solutions for cloud environments.

- Implement Infrastructure as Code (IaC) using tools like Terraform, AWS CloudFormation, or CDK.

Data Solutions with Databricks:

- Serve as a Databricks SME, guiding the architecture and implementation of ETL pipelines, data lakes, and machine learning workflows using Databricks on AWS.

- Optimize Databricks clusters for performance, cost, and security.

- Integrate Databricks with other AWS services like S3, Redshift, and Glue for seamless data processing.

Leadership and Team Collaboration:

- Lead a team of cloud engineers, providing mentorship and technical guidance.

- Collaborate with cross-functional teams including data scientists, analysts, and developers to understand requirements and translate them into scalable cloud and data architectures.

- Manage project timelines, ensuring timely delivery of cloud and data solutions.

Security and Compliance:

- Implement best practices for cloud security, including identity and access management (IAM), encryption, and VPC security.

- Ensure compliance with industry standards (e.g., GDPR, HIPAA) and internal security policies.

- Conduct regular security audits and optimize cloud configurations to reduce risk.

Performance Optimization and Monitoring:

- Set up monitoring, logging, and alerting using AWS CloudWatch, Datadog, or similar tools.

- Analyze system performance and provide recommendations for improvement.

- Troubleshoot complex issues related to cloud infrastructure, data pipelines, and Databricks clusters.

Technical Skills:

- Strong expertise in AWS services (EC2, S3, Lambda, RDS, VPC, CloudFormation, IAM, etc.).

- Proficiency with Databricks, including building and managing data pipelines, data lakes, and machine learning models.

- Experience with Infrastructure as Code (IaC) tools like Terraform or AWS CloudFormation.

- Knowledge of big data technologies like Apache Spark, Hadoop, or Delta Lake.

- Proficient in programming and scripting languages (e.g., Python, SQL, Scala).

- Experience with CI/CD pipelines for deploying cloud infrastructure.

Soft Skills:

- Strong leadership and project management skills.

- Excellent problem-solving and analytical skills.

- Ability to work collaboratively with diverse teams.

- Strong communication skills, with the ability to explain technical concepts to non-technical stakeholders.

Certifications:

Required:

- AWS Certified Solutions Architect (Associate or Professional)

- Databricks Certified Associate Developer for Apache Spark

Preferred:

- AWS Certified DevOps Engineer Professional

- AWS Certified Big Data Specialty or AWS Certified Data Analytics Specialty

- Databricks Certified Professional Data Engineer

Job Tags

Similar Jobs