HiPEAC2023-DL4IoT Workshop_Jean Hagemeyer presentation
•Download as PPTX, PDF•
0 likes•5 views
VEDLIOT – Accelerated AIoT. Jens Hagemeyer. 2nd Workshop on Deep Learning for IoT (DL4IoT), co-located with HiPEAC 2023, Toulouse, France, January 2023
Report
Share
More Related Content
Similar to HiPEAC2023-DL4IoT Workshop_Jean Hagemeyer presentation
Similar to HiPEAC2023-DL4IoT Workshop_Jean Hagemeyer presentation (20)
More from VEDLIoT Project
More from VEDLIoT Project (17)
Recently uploaded
Recently uploaded (20)
HiPEAC2023-DL4IoT Workshop_Jean Hagemeyer presentation
- 1. Jens Hagemeyer Bielefeld University VEDLIoT – Accelerated AIoT
- 2. 2 Platform Hardware: Scalable, heterogeneous, distributed Accelerators: Efficiency boost by FPGA and ASIC technology Toolchain: Optimizing Deep Learning for IoT Use cases Industrial IoT Automotive Smart Home Open call 10 projects covering a wide range of AIoT applications Early use and evaluation of VEDLIoT technology Very Efficient Deep Learning for IoT – VEDLIoT Call: H2020-ICT2020-1 Topic: ICT-56-2020 Next Generation Internet of Things Duration: 1. November 2020 – 31. Oktober 2023 Coordinator: Bielefeld University (Germany) Overall budget: 7 996 646.25 € Consortium: 12 partners from 4 EU countries (Germany, Poland, Portugal and Sweden) and one associated country (Switzerland). More info: https://www.vedliot.eu/ https://twitter.com/VEDLIoT https://www.linkedin.com/company/vedliot/
- 4. 4 VEDLIoT Hardware Platform Heterogeneous, modular, scalable microserver system Supporting the full spectrum of IoT from embedded over the edge towards the cloud Different technology concepts for improving x86 GPU ML-ASIC ARM v8 GPU SoC FPGA SoC RISC-V FPGA VEDLIOT Cognitive IoT Platform Performance Cost-effectiveness Maintainability Reliability Energy-Efficiency Safety
- 5. 5 RECS Architecture – RECS|BOX RECS Server Backplane (up to 15 Carriers) Carrier (PCIe Expansion) Carrier (High Performance) e.g. GPU-Accelerator Carrier (Low Power) #3 #2 Microserver (High Performance) #1 Microserver (Low Power) #16 #3 #2 Microserver (Low Power) #1 High-Speed Low-Latency Network (PCIe, High-Speed Serial) Compute Network (up to 40 GbE) Management Network (KVM, Monitoring, …) HDMI/USB iPass+ HD QSFP+ RJ45 Ext. Connectors GPU SoC FPGA SoC ARM Soc Low-Power Microserver (Apalis/Jetson) x86 ARM v8 High-Performance Microserver (COM Express) FPGA SoC High-Performance Carrier (up to 3 microservers) Low-Power Carrier (up to 16 microservers)
- 6. 6 t.RECS t.RECS Edge Server Optimized platform for local / edge applications Provide interfaces for Video Camera Peripheral input (USB) Combine FPGA and GPU acceleration Compact dimensions 1 RU, E-ATX form factor (2 RU/ 3 RU for special cases) RECS Architecture – t.RECS Microserver #3 (COM-HPC Client) Microserver #1 (COM-HPC Client) Microserver #2 (COM-HPC Server) Switched PCIe (Host to Host) External interfaces PCIe expansion Ethernet (up to 10 GbE) Management Network (KVM, Monitoring, …) I/O (Camera, Display, Radar/Lidar, Audio)
- 7. 7 u.RECS u.RECS AIoT Server Supports ML acceleration FPGA ASIC Communication interfaces Wired (CAN, Ethernet, CSI) Wireless (WLAN, LoRa, 5G) Sensors Camera Environment (Temp./Hum.) Housekeeping Embedded Device (~ 20x20x6 cm) RECS Architecture – u.RECS PCIe Ethernet (1 GbE & SPE) Management & Monitoring I/O (Camera, WiFi, LoRa, 4G/5G) Microserver #1 (SMARC 2.1) Microserver #2 (Jetson NX) ML Acc. (M.2) Front Panel 2x HDMI RJ45/ SPE 4x USB 3.1
- 9. 9 Peak performance values of specialized accelerators, provided by the vendors (precisions varying from INT8 to FP32) Peak Performance of DL Accelerators Average efficiency at 1000 GOPS /W [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] 1 10 100 1,000 10,000 100,000 1,000,000 10,000,000 0.01 0.1 1 10 100 1000 Performance [GOPS] Power [Watt] ASIC GPU FPGA Ultra Low Power High Performance Low Power
- 10. 10 Yolo v4 accelerator performance Performance of Yolo v4 for different hardware platform has been evaluated Performance measurement for other networks (Resnet, EfficientNet) available as well
- 11. 11 ▪ VEDLIoT accelerators support a large variety of reconfigurable architectures ▪ From small embedded FPGAs to large ACAPs ▪ Large design space for FPGA-based accelerators ▪ Dynamic hardware reconfiguration ▪ Adapt to changing requirements at run-time ▪ Change characteristics of DL-accelerator ▪ Trade-off between power and performance, power and accuracy, etc. ▪ Inference and training on FPGA ▪ Supports quantization from int8 to float32 ▪ DL and Deep Reinforcement Learning Reconfigurable DL accelerators
- 12. 12 DL accelerator co-design "FiBHA: Fixed Budget Hybrid CNN Accelerator", Fareed Qararyah, Muhammad Waqar Azhar, Pedro Trancoso, IEEE 34th International Symposium on Computer Architecture and High- Performance Computing (SBAC-PAD 2022), Bordeaux, France, November 2–5 2022 Monolithic design ● One engine computes all the core layers ● E.g. TPU SEML ● One engine computes all layers of the same type ● PW engine, DW engine SESL ● One engine per layer ● E.g. FINN FiBHA ● SESL + SEML
- 13. 13 Memory management for DL accelerators ▪ RAINBOW tool ▪ Different types of memory buffer strategies ▪ Different types of optimizers ▪ Layer-by-layer on chip memory analysis ▪ Requirements determine best heterogeneous execution plan (combination of different strategies)
- 14. 14 ▪ Common environment for running distributed applications ▪ WebAssembly runtime + Trusted Execution Environment ▪ Security for edge (and cloud) devices ▪ Advances on attestation ▪ Better support for edge devices ▪ Distributed (Byzantine fault-tolerant) attestation and configuration service ▪ Secure IoT Gateway Security
- 15. 15 Simulation platform for ML accelerators ▪ RISC-V SoCs and Custom Function Units ▪ Improve test and verification ▪ Co-simulate Verilog blocks ▪ Used in Google’s CFU Playground ▪ Continuous integration based in Gitlab and Google Cloud Platform Safety and Robustness Robustness verification on DL models ▪ Tuning hyperparameters
- 16. 16 A compositional architecture framework for AIoT Knowledge creation (e.g. definition of safety goals). Concept design (e.g. introduction of redundancy to fulfil safety goals). Final design (e.g. assigning functions to independent processors to guarantee redundancy). Monitoring concept definition (e.g. monitoring fulfilment of safety goals at run-time). Solution Space Problem Space
- 17. 17 ▪ Focus on collision detection/avoidance scenario ▪ Improve performance/cost ratio – AI processing hardware distributed over the entire chain Use case: Automotive Challenge: Distribution of work
- 18. 18 ▪ Control applications need DL-based condition classification ▪ On the edge device for low power consumption ▪ Suggestions for control and maintenance ▪ DL methods on all communication layers ▪ DL in a distributed architecture ▪ Dynamically configured systems ▪ Sensored testbench with 2 motors ▪ Acceleration, Magnetic field, Temperature, IR-Cam (temperature), Current-Sensors, Torque Use case: Industrial IoT – drive condition classification ▪ On / Off detection without motor current or voltage ▪ Cooling fault detection ▪ Bearing fault detection Challenge: Low-power / Efficiency
- 19. 19 Use case: Industrial IoT – Arc detection ▪ AI based pattern recognition for different local sensor data ▪ current, magnetic field, vibration, temperature, low resolution infrared picture ▪ Safety critical nature ▪ response time should be <10ms ▪ AI based or AI supported decision made by the sensor node itself or by a local part of the sensor network Challenge: Accuracy
- 20. 20 ▪ Face recognition ▪ Mobilenet SSD trained on WIDERFACE dataset ▪ Object detection ▪ YoloV3, Efficient-Net, yoloV4-tiny ▪ Gesture detection ▪ YoloV4-tiny with 3 Yolo layers (usually: 2 layers) ▪ Speech recognition ▪ Mozilla DeepSpeech ▪ AI Art: Style-Gan trained on works of arts ▪ Collect usage data in situation memory Use case: Smart Mirror – Neural Networks Challenge: Data privacy, Efficiency
- 21. 21 Thank you for your attention. Contact Jens Hagemeyer, Carola Haumann Bielefeld University, Germany chaumann@cor-lab.uni-bielefeld.de jhagemey@cit-ec.uni-bielefeld.de
- 22. 22 Bielefeld University (UNIBI) - Coordinator Christmann (CHR) University of Osnabrück (UOS) Siemens (SIEMENS) University of Neuchâtel (UNINE) University of Lisbon (FC.ID) Chalmers (CHALMERS) University of Gothenburg (UGOT) RISE (RISE) EmbeDL (EMBEDL) Veoneer (VEONEER) Antmicro (ANT) Partners
- 23. 23 ▪ Increase safety, health and well being of residents – acceleration of AI methods for demand-oriented user-home interaction ▪ Smart Mirror as central user interface ▪ Own mirror image can be seen normally ▪ Intuitive control over gesture and voice ▪ Shows personalized information ▪ Data privacy as the highest priority ▪ Edge computation of many neural networks Use case: Smart Home / Assisted Living
- 24. 24 VEDLIoT Deep Learning Plattforms Supported Computer-On-Module form factors Raspberry Pi Compute Module 4 Jetson Xavier NX SMARC Xilinx Kria Jetson AGX Xavier COM Express (Type 6/7) COM-HPC Client (Type A-C) COM-HPC Server (Type D/E) Size (higher distance is smaller) I/O Flexibility Performance Supported Architectures Market Share uRECS RECS|Box & t.RECS
- 25. 25 Benchmark performance of DL accelerators Comparison based on currently available architectures VEDLIoT will include new specialized accelerators 0 50 100 150 200 250 300 350 Coral (M.2) Coral (Dev.) Xavier AGX (LP) Xavier AGX (HP) Xavier NX TX2 Nano GTX1660 ZU15 ZU3 Xeon-D1577 Epyc3451 Myriad GAP8 Energy Efficiency [GOPS/W] ResNet50 Int 8 ResNet50 FP16 ResNet50 FP32 YoloV4 Int 8 YoloV4 FP16 YoloV4 FP32 MobileNet Int 8 MobileNet FP16 MobileNet FP32
- 26. 26 Flexible Accelerators for Deep Learning DL Model DL Model CPU, GPU- SoC, ML-SoC FPGA-SoC End of Moore’s law & dark silicon => Domain Specific Architectures (DSA) Efficient, flexible, scalable accelerators for the compute continuum Algotecture Optimized DL algorithms Optimized toolchain Optimized computer architecture Heterogeneous DL Accelerator Algotecture/ Co-Designed DL Accelerator Compiler Co-Design
- 27. 27 VEDLIoT‘s Deep Learning Toolchain • Image Classification • Object Detection • Semantic Segmentation • Instance Segmentation • Extractive Question Answering Model Zoo Optimization Engine Compilers & Runtime APIs Heterogeneous Hardware Platforms
- 28. 28 Benchmark performance of DL accelerators YoloV4 [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLR… [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRA… [CELLRANGE] 10 100 1000 10000 2 4 8 16 32 64 128 Performance [GOPS] Power [Watt] INT8 FP16 FP32
- 29. 29 Benchmark performance of DL accelerators ResNet50 [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELL… [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] 10 100 1000 10000 2 4 8 16 32 64 128 Performance [GOPS] Power [Watt] INT8 FP16 FP32
- 30. 30 [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CEL… [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] [CELLRANGE] 10 100 1000 10000 2 4 8 16 32 64 128 Performance [GOPS] Power [Watt] INT8 FP16 FP32 Benchmark performance of DL accelerators MobileNetV3