AWS Data Engineer

<div><strong style="color: rgb(239, 72, 21);">Role and responsibilities</strong><span style="color: rgb(239, 72, 21);">&nbsp;</span></div><div class="ql-align-justify">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: black;">Understands the process flow and the impact on the project module outcome.&nbsp;</span></div><div class="ql-align-justify">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: black;">Works on coding assignments for specific technologies basis the project requirements and documentation available</span></div><div class="ql-align-justify">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: black;">Debugs basic software components and identifies code defects.&nbsp;</span></div><div class="ql-align-justify">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: black;">Focusses on building depth in project specific technologies.&nbsp;</span></div><div class="ql-align-justify">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: black;">Expected to develop domain knowledge along with technical skills.&nbsp;</span></div><div class="ql-align-justify">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: black;">Effectively communicate with team members, project managers and clients, as required.&nbsp;</span></div><div class="ql-align-justify">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span style="color: black;">A proven high-performer and team-player, with the ability to take the lead on projects.&nbsp;</span></div><div class="ql-align-justify"><span style="color: black;">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Design and create S3 buckets and folder structures (raw, cleansed_data, output, script, temp-dir, spark-ui)</span></div><div class="ql-align-justify"><span style="color: black;">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Develop AWS Lambda functions (Python/Boto3) to download Bhav Copy via REST API and ingest into S3</span></div><div class="ql-align-justify"><span style="color: black;">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Author and maintain AWS Glue Spark jobs to:</span></div><div class="ql-align-justify"><span style="color: black;">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;– partition data by scrip, year and month</span></div><div class="ql-align-justify"><span style="color: black;">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;– convert CSV to Parquet with Snappy compression</span></div><div class="ql-align-justify"><span style="color: black;">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Configure and run AWS Glue Crawlers to populate the Glue Data Catalog</span></div><div class="ql-align-justify"><span style="color: black;">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Write and optimize AWS Athena SQL queries to generate business-ready datasets</span></div><div class="ql-align-justify"><span style="color: black;">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Monitor, troubleshoot and tune data workflows for cost and performance</span></div><div class="ql-align-justify"><span style="color: black;">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Document architecture, code and operational runbooks</span></div><div class="ql-align-justify"><span style="color: black;">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Collaborate with analytics and downstream teams to understand requirements and deliver SLAs</span></div><div class="ql-align-justify">&nbsp;</div><div><strong style="color: rgb(239, 72, 21);">&nbsp;</strong></div><div><strong style="color: rgb(239, 72, 21);">Technical&nbsp;skills&nbsp;requirements</strong><span style="color: rgb(239, 72, 21);">&nbsp;</span></div><div><span style="color: black;">&nbsp;</span></div><div><span style="color: black;">The candidate must demonstrate proficiency in,</span></div><div><span style="color: rgb(36, 36, 36);">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3+ years’ hands-on experience with AWS data services (S3, Lambda, Glue, Athena)</span></div><div><span style="color: rgb(36, 36, 36);">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;PostgreSQL basics</span></div><div><span style="color: rgb(36, 36, 36);">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Proficient in SQL and data partitioning strategies</span></div><div><span style="color: rgb(36, 36, 36);">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Experience with Parquet file formats and compression techniques (Snappy)</span></div><div><span style="color: rgb(36, 36, 36);">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Ability to configure Glue Crawlers and manage the AWS Glue Data Catalog</span></div><div><span style="color: rgb(36, 36, 36);">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Understanding of “serverless” architecture and best practices in security, encryption and cost control</span></div><div><span style="color: rgb(36, 36, 36);">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Good documentation, communication and problem-solving skills&nbsp;</span></div><div><span style="color: black;">&nbsp;</span></div><div><strong style="color: rgb(239, 72, 21);">&nbsp;</strong></div><div><strong style="color: rgb(239, 72, 21);">Nice-to-have skills</strong><span style="color: black;">&nbsp;</span></div><div><span style="color: black;">&nbsp;</span></div><div class="ql-align-justify">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;SQL Database </div><div class="ql-align-justify">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Experience in Python (or “Rubby”) scripting to integrate with AWS services</div><div class="ql-align-justify">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Familiarity with RESTful API consumption and JSON processing</div><div class="ql-align-justify">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Background in financial markets or working with large-scale time-series data</div><div class="ql-align-justify">·&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Knowledge of CI/CD pipelines for data workflows</div><div><br></div>

Khushbu

business_unit

client_name

Sourcing

Shortlisted

Applied

Tracy Interviewed

Recruiter Check

Quality Check

Pending Business Review

L1 to be scheduled

L1 Interview

L1 Shortlisted

L2 to be scheduled

L2 Interview

L2 Shortlisted

Client

Submmitted to Client

Client round-1 to be scheduled

Client round-1 Interview

Client round-2 to be scheduled

Client round-2 Interview

Hired

Offered

Joined

Offer Rejected

On Hold