IMPROVEMENT OF SPARK STREAMING THROUGH BATCH SIZING WITH PERFORMANCE ANALYSIS
| Author(s) | : | Miss Urvashi Damor, Dr. D.A.Parikh |
| Institution | : | M.E. Student, Computer engineering, L.D. College of Engineering, Gujarat, India |
| Published In | : | Vol. 5, Issue 5 — May 2018 |
| Page No. | : | 48-52 |
| Domain | : | Engineering |
| Type | : | Research Paper |
| ISSN (Online) | : | 2348-4470 |
| ISSN (Print) | : | 2348-6406 |
The need for real-time processing of “big-data” has led to the development of framework for distributedstream processing in cluster. It is important for such framework to be robust against variable operating conditions suchas server failures, changes in data ingestion rates, and workload characteristics.To provide fault tolerance and efficient stream processing at scale, recent stream processing framework have proposedto treat streaming workloads as a series of batch jobs on small batches of streaming data. The robustness of suchframework against variable operating conditions has not been explored.We explore the effects of the batch size on the performance of streaming workloads. The throughput and end-to-endlatency of the system can have complicated relationships with batch sizes, data ingestion rates, variations in availableresources, workload characteristics, etc. We propose a simple yet robust control algorithm that automatically adapts thebatch size as the situation necessitates.
Miss Urvashi Damor, Dr. D.A.Parikh, “IMPROVEMENT OF SPARK STREAMING THROUGH BATCH SIZING WITH PERFORMANCE ANALYSIS”, International Journal of Advance Engineering and Research Development (IJAERD), Vol. 5, Issue 5, pp. 48-52, May 2018.
