put data? 로 받는 방법이 6가지 있는데
kinesis data stream으로 받는방법, iot는 사용하지 않을 예정이라 뺌
1. firehose Agent
2. AWS sdk
3. cloudwatch logs
4. cloudwatch events
agent는 내부에 설치하고 연결
2. aws sdk 이거는 어떻게 하는지 잘 모르겠음
3. cloudwatch logs 로그그룹에서 구독필터 사용하여 연결(내부에 cloudwatch agent 설치해야함/ 사용요금이 증가한다는데..)
4. cloud watch events
CloudWatch Events (eventbridge)규칙에 대상을 추가해 Kinesis Data Firehose 전송 스트림으로 이벤트를 보내도록 Amazon CloudWatch를 구성할 수 있습니다.
Dynamic partitioning
이게 뭐야
ec2, firehose agent를 연결
https://qiita.com/meta_plankton/items/4e47ca1f76ae2c80207d
EC2のログ→Kinesis firehose→S3を実装してみた - Qiita
これは何 EC2でログ生成→Kinesis firehose→S3、の流れを実装してみたの記事です。 Kinesis Data Firehoseを作成 コンソールからKinesisへ移動します。 配信ストリームを押下。 ...
qiita.com
firehose to s3 및 정리
https://blog.voidmainvoid.net/295
AWS kinesis Firehose 실습 및 예제 코드(Java), 장단점, 가격
실시간으로 비디오 및 데이터 스트림을 손쉽게 수집, 처리 및 분석 솔루션 Amazon Kinesis를 사용하면 실시간 스트리밍 데이터를 손쉽게 수집, 처리 및 분석할 수 있으므로 적시에 통찰력을 확보하
blog.voidmainvoid.net
cloudformation firehose
2019년도에 만든거라 좀...
GitHub - aws-samples/aws-research-workshops: This repo provides a managed SageMaker jupyter notebook with a number of notebooks
This repo provides a managed SageMaker jupyter notebook with a number of notebooks for hands on workshops in data lakes, AI/ML, Batch, IoT, and Genomics. - GitHub - aws-samples/aws-research-worksh...
github.com
datastream to glue s3
https://aws.amazon.com/ko/blogs/big-data/crafting-serverless-streaming-etl-jobs-with-aws-glue/
Crafting serverless streaming ETL jobs with AWS Glue | Amazon Web Services
Organizations across verticals have been building streaming-based extract, transform, and load (ETL) applications to more efficiently extract meaningful insights from their datasets. Although streaming ingest and stream processing frameworks have evolved o
aws.amazon.com
이건 iot 소스로
https://aws.amazon.com/ko/blogs/aws/new-serverless-streaming-etl-with-aws-glue/
New – Serverless Streaming ETL with AWS Glue | Amazon Web Services
When you have applications in production, you want to understand what is happening, and how the applications are being used. To analyze data, a first approach is a batch processing model: a set of data is collected over a period of time, then run through a
aws.amazon.com
high volume
Designing a High-volume Streaming Data Ingestion Platform Natively on AWS | Amazon Web Services
The total global data storage is projected to exceed 200 zettabytes by 2025. This exponential growth of data demands increased vigilance against cybercrimes. Emerging cybersecurity trends include increasing service attacks, ransomware, and critical infrast
aws.amazon.com
https://techmagie.wordpress.com/2019/07/29/aws-kinesis-for-logs-data-ingestion/
AWS Kinesis for logs data ingestion
This is third blog of series “People Analytics and Attrition prediction using Serverless architecture and AI/ML” . In first blog I talked about Serverless Architecture,People Analytics …
techmagie.wordpress.com
Creating an AWS Data Lake and Real-Time Pipeline for Mobile App Analytics
IntroductionOrganizations that successfully generate business value from their data will outperform their peers. These leaders are able to do new types of analytics like machine learning over new sources like log files, data from click-streams, social medi
www.xerris.com
firehose vs datastream 차이
[AWS] Kinesis Data Streams / Firehose 비교
Amazon Kinesis services collect, process, and analyze data streams in real time. AWS Kinesis는 AWS에서 제공하는 완전관리형 실시간 스트리밍 데이터 처리 서비스로 기능에 따라 Kinesis Video Streams, Kin..
jellybean.tistory.com
splunk 다운
https://www.splunk.com/ko_kr/download.html
다운로드 | Splunk
Splunk가 조직에 제공할 수 있는 이점을 직접 확인해 보십시오. Splunk 소프트웨어는 모든 사람이 액세스할 수 있고 모든 사람에게 유용하고 가치 있는 머신 데이터를 제공합니다. 무료로 Splunk를 사
www.splunk.com
가이드 aws랑 같은것같음
How to stream AWS CloudWatch Logs to Splunk (Hint: it’s easier than you think)
See step-by-step process of AWS Lambda streaming of CloudWatch Logs into Splunk for real-time security analysis, visualization, troubleshooting.
www.splunk.com
이건 개빡센거 같음
https://catalog.us-east-1.prod.workshops.aws/workshops/2300137e-f2ac-4eb9-a4ac-3d25026b235f/en-US
Workshop Studio
catalog.us-east-1.prod.workshops.aws
'CLOUD > AWS' 카테고리의 다른 글
| AWS glue 자습서 (0) | 2022.06.16 |
|---|---|
| [awsblog]exponential backoff (0) | 2022.06.15 |
| cloudwatch log flow 의 iam 역할 (0) | 2022.06.14 |
| react 앱 s3에 폴더별로 설치 방법(403 에러 해결) (0) | 2022.06.14 |
| 하나의 S3 버킷에 여러 App(web)을 Cloudfront 에 띄우기 및 루트 설정 (1) | 2022.06.13 |
댓글