On-premises data is stored in various internal systems. If the customer has data on AWS already, it can be consumed directly too. Use AWS Direct Connect or AWS Client VPN for connectivity. Optional if data sources are already on AWS. SamyaData Collector (StreamSetsrunning on Amazon Elastic Compute Cloud (Amazon EC2) is used to connect to various sources to fetch the raw data and store it in SamyaRaw Data Lake on Amazon Simple Storage Service (Amazon S3). AWS Glue is used for data cataloging along with AmazonAthenaquery and filter andAmazon EMRis used for data pre-processing. Processed data is stored in SamyaProcessed Data Lake on AmazonS3. Publicly available external data is fetched by Apache Kafka running on AmazonEC2 with events to trigger AWS Lambda whichsends data to AmazonKinesis Data Firehosefor storing it in a SamyaRaw Data Lake. External dataset is also pre-processed by Spark MLlib on AmazonEC2and stored in a SamyaProcessed Data Lake. TensorFlow and PyTorchAI/ML models run on Amazon EKSwith Kubeflow for forecasting Delta Lake running on AmazonEC2 manages update and deletion in datasets, data versioning and metadata handling in the SamyaProcessed Data Lake. SamyaData Writer (StreamSetsrunning on Amazon EC2) writes the AI/ML output to Amazon Relational Database Service (Amazon RDS) and the model is stored back in the SamyaProcessed Data
