More Related Content
Similar to Major Outages in Major Enterprises Payara Conference
Similar to Major Outages in Major Enterprises Payara Conference (20)
More from Tier1 app
More from Tier1 app (20)
Recently uploaded
Recently uploaded (20)
Major Outages in Major Enterprises Payara Conference
- 1. Ram Lakshmanan | architect: yCrash Major Outages in Major Enterprises
- 2. 1. Unresponsiveness in middleware Major Financial Institution in N. America
- 3. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump (optional) 360-degree data 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H Open-source script: https://github.com/ycrash/yc-data-script ./yc –p <PROCESS_ID>
- 4. 2019-12-26 17:13:23 Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.7-b01 mixed mode): "Reconnection-1" prio=10 tid=0x00007f0442e10800 nid=0x112a waiting on condition [0x00007f042f719000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x007b3953a98> (a java.util.concurrent.locks.AbstractQueuedSynchr) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.lang.Thread.run(Thread.java:722) : : 1 2 3 1 Timestamp at which thread dump was triggered 2 JVM Version info 3 Thread Details - <<details in following slides>> Anatomy of thread dump "InvoiceThread-A996" prio=10 tid=0x00002b7cfc6fb000 nid=0x4479 runnable [0x00002b7d17ab8000] java.lang.Thread.State: RUNNABLE at com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380) at com.buggycompany.rt.util.ItinerarySegmentProcessor.processTripType0(ItinerarySegmentProcessor.java:366) at com.buggycompany.rt.util.ItinerarySegmentProcessor.processItineraryByTripType(ItinerarySegmentProcessor.java:254) at com.buggycompany.rt.util.ItinerarySegmentProcessor.templateMethod(ItinerarySegmentProcessor.java:399) at com.buggycompany.qc.gds.InvoiceGeneratedFacade.readTicketImage(InvoiceGeneratedFacade.java:252) at com.buggycompany.qc.gds.InvoiceGeneratedFacade.doOrchestrate(InvoiceGeneratedFacade.java:151) at com.buggycompany.framework.gdstask.BaseGDSFacade.orchestrate(BaseGDSFacade.java:32) at com.buggycompany.framework.gdstask.BaseGDSFacade.doWork(BaseGDSFacade.java:22) at com.buggycompany.framework.concurrent.BuggycompanyCallable.call(buggycompanyCallable.java:80) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722)
- 5. "InvoiceThread-A996" prio=10 tid=0x00002b7cfc6fb000 nid=0x4479 runnable [0x00002b7d17ab8000] java.lang.Thread.State: RUNNABLE at com.buggycompany.rt.util.ItinerarySegmentProcessor.setConnectingFlight(ItinerarySegmentProcessor.java:380) at com.buggycompany.rt.util.ItinerarySegmentProcessor.processTripType0(ItinerarySegmentProcessor.java:366) at com.buggycompany.rt.util.ItinerarySegmentProcessor.processItineraryByTripType(ItinerarySegmentProcessor.java:254) at com.buggycompany.rt.util.ItinerarySegmentProcessor.templateMethod(ItinerarySegmentProcessor.java:399) at com.buggycompany.qc.gds.InvoiceGeneratedFacade.readTicketImage(InvoiceGeneratedFacade.java:252) at com.buggycompany.qc.gds.InvoiceGeneratedFacade.doOrchestrate(InvoiceGeneratedFacade.java:151) at com.buggycompany.framework.gdstask.BaseGDSFacade.orchestrate(BaseGDSFacade.java:32) at com.buggycompany.framework.gdstask.BaseGDSFacade.doWork(BaseGDSFacade.java:22) at com.buggycompany.framework.concurrent.BuggycompanyCallable.call(buggycompanyCallable.java:80) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) 1 2 3 4 5 6 7 1 Thread Name - InvoiceThread-A996 2 Priority - Can have values from 1 to 10 3 Thread Id - 0x00002b7cfc6fb000 – Unique ID assigned by JVM. It's returned by calling the Thread.getId() method. 4 Native Id - 0x4479 - This ID is highly platform dependent. On Linux, it's the pid of the thread. On Windows, it's simply the OS-level thread ID within a process. On Mac OS X, it is said to be the native pthread_t value. 5 Address space - 0x00002b7d17ab8000 - 6 Thread State - RUNNABLE 7 Stack trace -
- 6. 2. Poor response time Major cloud service
- 7. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 360-degree data 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H Open-source script: https://github.com/ycrash/yc-data-script
- 9. How are objects Garbage Collected? Evolution: Manual -> Automatic 3 – 4 decades before Now Developer Writes code to Manually evict Garbage JVM Automatically evicts Garbage
- 10. Automatic GC sounds good right? Yes, but for Cpu consumption gc pauses Success Stories GC tuning
- 11. 2019-08-31T01:09:19.397+0000: 1.606: [GC (Metadata GC Threshold) [PSYoungGen: 545393K->18495K(2446848K)] 545393K- >18519K(8039424K), 0.0189376 secs] [Times: user=0.15 sys=0.01, real=0.02 secs] 2019-08-31T01:09:19.416+0000: 1.625: [Full GC (Metadata GC Threshold) [PSYoungGen: 18495K->0K(2446848K)] [ParOldGen: 24K- >17366K(5592576K)] 18519K->17366K(8039424K), [Metaspace: 20781K->20781K(1067008K)], 0.0416162 secs] [Times: user=0.38 sys=0.03, real=0.04 secs] 2019-08-31T01:18:19.288+0000: 541.497: [GC (Metadata GC Threshold) [PSYoungGen: 1391495K->18847K(2446848K)] 1408861K- >36230K(8039424K), 0.0568365 secs] [Times: user=0.31 sys=0.75, real=0.06 secs] 2019-08-31T01:18:19.345+0000: 541.554: [Full GC (Metadata GC Threshold) [PSYoungGen: 18847K->0K(2446848K)] [ParOldGen: 17382K->25397K(5592576K)] 36230K->25397K(8039424K), [Metaspace: 34865K->34865K(1079296K)], 0.0467640 secs] [Times: user=0.31 sys=0.08, real=0.04 secs] 2019-08-31T02:33:20.326+0000: 5042.536: [GC (Allocation Failure) [PSYoungGen: 2097664K->11337K(2446848K)] 2123061K- >36742K(8039424K), 0.3298985 secs] [Times: user=0.00 sys=9.20, real=0.33 secs] 2019-08-31T03:40:11.749+0000: 9053.959: [GC (Allocation Failure) [PSYoungGen: 2109001K->15776K(2446848K)] 2134406K- >41189K(8039424K), 0.0517517 secs] [Times: user=0.00 sys=1.22, real=0.05 secs] 2019-08-31T05:11:46.869+0000: 14549.079: [GC (Allocation Failure) [PSYoungGen: 2113440K->24832K(2446848K)] 2138853K- >50253K(8039424K), 0.0392831 secs] [Times: user=0.02 sys=0.79, real=0.04 secs] 2019-08-31T06:26:10.376+0000: 19012.586: [GC (Allocation Failure) [PSYoungGen: 2122496K->25600K(2756096K)] 2147917K- >58149K(8348672K), 0.0371416 secs] [Times: user=0.01 sys=0.75, real=0.04 secs] 2019-08-31T07:50:03.442+0000: 24045.652: [GC (Allocation Failure) [PSYoungGen: 2756096K->32768K(2763264K)] 2788645K- >72397K(8355840K), 0.0709641 secs] [Times: user=0.16 sys=1.39, real=0.07 secs] 2019-08-31T09:04:21.406+0000: 28503.616: [GC (Allocation Failure) [PSYoungGen: 2763264K->32768K(2733568K)] 2802893K- >83469K(8326144K), 0.0789178 secs] [Times: user=0.12 sys=1.59, real=0.08 secs] Sample GC log
- 12. 3. Sudden CPU spike Major Automobile manufacturer
- 13. Healthy Saw-tooth GC pattern
- 14. Acute Memory Leak Pattern
- 15. Memory Leak GC pattern
- 16. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 360-degree data 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H Open-source script: https://github.com/ycrash/yc-data-script
- 17. 4. CPU spike Major Trading App
- 18. How to troubleshoot CPU spike? https://blog.fastthread.io/2020/04/23/troubleshooting-cpu-spike-in-a-major-trading-application/
- 19. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 360-degree data 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H Open-source script: https://github.com/ycrash/yc-data-script
- 20. What is ‘top -H’ data? top –h –p <PROCESS_ID>
- 21. 5. Degradation in response time Major Travel Service Provider
- 22. public void synchronized getData() { makeDBCall(); } Thread 1: Runnable Thread 2: BLOCKED Thread 1: Runnable BLOCKED thread state BLOCKED THREADS
- 23. 6. Intermittent HTTP 502 error In AWS EBS service
- 24. EBS Architecture
- 26. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 360-degree data 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H Open-source script: https://github.com/ycrash/yc-data-script
- 28. If you want to learn more… https://ycrash.io/java-performance-training Online-Training: Java Performance/Troubleshooting Master class @tier1app https://www.linkedin.com/company/ycrash This deck will be published in: https://blog.ycrash.io
Editor's Notes
- http://localhost:8080/yc-report.jsp?ou=SAP&de=198.134.23.1&app=yc&ts=2023-06-11T22-56-32
- http://localhost:8080/yc-report.jsp?ou=SAP&de=198.134.23.1&app=yc&ts=2023-06-11T22-56-32
- http://localhost:8080/yc-report.jsp?ou=SAP&de=198.134.23.1&app=yc&ts=2023-06-11T22-56-32
- Baseline: http://localhost:8080/yc-load-report-gc?ou=SAP&de=145.23.82.1&app=yc&ts=2023-06-11T23-03-50 Benchmark: http://localhost:8080/yc-load-report-gc?ou=SAP&de=193.45.89.12&app=yc&ts=2023-06-11T23-09-10
- Baseline: http://localhost:8080/yc-load-report-gc?ou=SAP&de=145.23.82.1&app=yc&ts=2023-06-11T23-03-50 Benchmark: http://localhost:8080/yc-load-report-gc?ou=SAP&de=193.45.89.12&app=yc&ts=2023-06-11T23-09-10
- Baseline: http://localhost:8080/yc-load-report-gc?ou=SAP&de=145.23.82.1&app=yc&ts=2023-06-11T23-03-50 Benchmark: http://localhost:8080/yc-load-report-gc?ou=SAP&de=193.45.89.12&app=yc&ts=2023-06-11T23-09-10
- http://localhost:8080/yc-load-report-ft?ou=SAP&de=32.123.89.12&app=yc&ts=2023-06-11T23-54-10
- http://localhost:8080/yc-load-report-ft?ou=SAP&de=32.123.89.12&app=yc&ts=2023-06-11T23-54-10
- http://localhost:8080/yc-load-report-ft?ou=SAP&de=32.123.89.12&app=yc&ts=2023-06-11T23-54-10
- https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjIvMDcvMzEvdGhyZWFkX2thc3RsZV8yNjA3MjIudHh0LS03LTMwLTMzLS0xNi0zMy0zNg==&s=t
- https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjIvMDcvMzEvdGhyZWFkX2thc3RsZV8yNjA3MjIudHh0LS03LTMwLTMzLS0xNi0zMy0zNg==&s=t
- http://172.19.248.49:8080/yc-load-report-ft?ou=SAP&de=64.12.89.33&app=yc&ts=2023-06-12T00-07-59
- http://localhost:8080/yc-report.jsp?ou=SAP&de=198.134.23.1&app=yc&ts=2023-06-11T22-56-32