Job Description
Who we are
Artmac Soft is a technology consulting and service-oriented IT company dedicated to providing innovative technology solutions and services to Customers.
Job Description:
Job Title :
Senior Big Data Engineer - Python, Spark, Kafka
Job Type : W2 / C2C
Experience : 8-15 Years
Location : Celebration, Florida / Santa Clara, California
Responsibilities:
• Minimum 7 years of related work experience
• Advanced development skills in cross-platform coding languages such as SQL/TSQL, Python, Java, and VBA
• SQL and scripting expertise for data extraction, transformation, and building performant datasets, with a focus on automation
• Extensive experience with relational databases and NoSQL databases (e.g., Snowflake, SQL Server, Oracle, MongoDB)
• Comfortable working in an agile environment, collaborating with cross-functional teams, and adapting to changing requirements
• KEY: Expert in MicroStrategy report and dashboard development, with a desire to further develop technical expertise in data engineering tools, methodologies, and emerging technologies.
• Queuing Technology – Kafka, RabbitMQ, Redis, SQS, Kinesis Streams, Kinesis Firehose
• Data Processing – EMR, Spark, Glue, Spark Streaming/Flink
• Containers - Docker, Docker Swarm, Docker Applications
• CICD - Jenkins/Codebuild/GitLab
• Security - IAM roles, wire encryption, KMS, Kerberos, Authz, AD
• Infrastructure as Code - Terraform, Cloud Formation, CDK
• Proficient in cloud technologies, including AWS (e.g., Lambda, Kinesis, DMS, AppFlow).
Preferred Skills and Experience:
- 8-10 years of data warehousing/engineering, data modeling, and software solutions design and development experience.
- Expertise in Python & PySpark for scripting, automation, and data engineering tasks [Must Have].
- Experience with big data processing frameworks like Apache Spark, EMR, Glue, or Flink [Must Have].
- Understanding of queueing and streaming technologies like Kafka, RabbitMQ, Redis, Kinesis [Must Have].
- Hands-on experience with security best practices, including Kerberos, IAM policies, and AD integration (Good to have).
- Strong hands-on experience with big data systems such as HDFS, hive, Impala, and Spark on Cloudera Hadoop distribution or Databricks
- Familiarity with MicroStrategy for reporting and dashboard development (a plus) [Good to have].
- Must be proficient in SPARK, KAFKA, and streaming technologies.
- Experience with an RDBMS –Microsoft SQL Server, Oracle, DB2 (nice to have).
Qualification:
- Bachelor's degree or equivalent combination of education and experience.