simple radar knowledge can easily be combined with complex data-driven learning Therefore, the NN marked with the red dot is not optimal w.r.t.the number of MACs. We find that deep radar classifiers maintain high-confidences for ambiguous, difficult samples, e.g. The proposed Our results demonstrate that Deep Learning methods can greatly augment the classification capabilities of automotive radar sensors. Our investigations show how The measurement scenarios should cover typical road traffic situations, as described by Euro NCAP, for more details see [18, 19]. Then, it is shown that this manual design process can be replaced by a neural architecture search (NAS) algorithm, which finds a CNN with similar accuracy, but with even less parameters. extraction of local and global features. In, the range-Doppler spectrum is computed for multiple cycles, and a combination of a CNN and Long-Short-Term-Memory (LSTM) neural network is used for a 2-class classification problem. View 3 excerpts, cites methods and background. available in classification datasets. automotive radar sensors,, R.Prophet, M.Hoffmann, A.Ossowska, W.Malik, C.Sturm, and There are many search methods in the literature, each with advantages and shortcomings. In contrast to these works, data-driven DL approaches learn a rich representation in an end-to-end training, such that no additional feature extraction is necessary. 1. integrated into an 24 ghz automotive radar, in, A.Bartsch, F.Fitzek, and R.Rasshofer, Pedestrian recognition using This work demonstrates a possible solution: 1) A data preprocessing stage extracts sparse regions of interest (ROIs) from the radar spectra based on the detected and associated radar reflections. The reflection branch was attached to this NN, obtaining the DeepHybrid model. The different versions of the original document can be found in: Volume 2019, 2019DOI: 10.1109/radar.2019.8835775Licence: CC BY-NC-SA license. It uses a chirp sequence-like modulation, with the difference that not all chirps are equal. M.Vossiek, Image-based pedestrian classification for 79 ghz automotive Experiments on a real-world dataset demonstrate the ability to distinguish relevant objects from different viewpoints. classifier architecture search, in, R.Q. Charles, H.Su, M.Kaichun, and L.J. Guibas, Pointnet: Deep Each track consists of several frames. to improve automatic emergency braking or collision avoidance systems. Object type classification for automotive radar has greatly improved with recent deep learning (DL) solutions, however these developments have mostly focused on the classification accuracy. The proposed method can be used for example On the other hand, if there is a small object that can be run over, e.g.a can of coke, the ego-vehicle should classify it correctly and just ignore it. The ROI is centered around the maximum peak of the associated reflections and clipped to 3232 bins, which usually includes all associated patches. Mentioning: 3 - Radar sensors are an important part of driver assistance systems and intelligent vehicles due to their robustness against all kinds of adverse conditions, e.g., fog, snow, rain, or even direct sunlight. An novel object type classification method for automotive applications which uses deep learning with radar reflections, which fills the gap between low-performant methods of handcrafted features and high-performsant methods with convolutional neural networks. The processing pipeline from the radar time signal to the part of the radar spectrum that is used as input to the NN is depicted in Fig. Available: , AEB Car-to-Car Test Protocol, 2020. smoothing is a technique of refining, or softening, the hard labels typically layer. 5) by attaching the reflection branch to it, see Fig. Automotive radar has shown great potential as a sensor for driver assistance systems due to its robustness to weather and light conditions, but reliable classification of object types in real time has proved to be very challenging. Here, we focus on the classification task and not on the association problem itself, i.e.the assignment of different reflections to one object. The range-azimuth spectra are used by a CNN to classify different kinds of stationary targets in. By clicking accept or continuing to use the site, you agree to the terms outlined in our. 4 (c). samples, e.g. Our results demonstrate that Deep Learning methods can greatly augment the classification capabilities of automotive radar sensors. Nevertheless, both models mistake some pedestrian samples for two-wheeler, and vice versa. A range-Doppler-like spectrum is used to include the micro-Doppler information of moving objects, and the geometrical information is considered during association. The proposed approach automatically captures the intricate properties of the radar returns in order to minimize false alarms and fuse information from both the time-frequency and range domains. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). , and associates the detected reflections to objects. To the best of our knowledge, this is the first time NAS is deployed in the context of a radar classification task. N.Scheiner, N.Appenrodt, J.Dickmann, and B.Sick, Radar-based road user Automotive radar has shown great potential as a sensor for driver assistance systems due to its robustness to weather and light conditions, but reliable classification of object types in real time has proved to be very challenging. It can be observed that NAS found architectures with similar accuracy, but with an order of magnitude less parameters. Label smoothing is a technique of refining, or softening, the hard labels typically available in classification datasets. 2) We propose a hybrid model (DeepHybrid) that jointly processes the objects spectrum (spectral ROI) and reflection attributes (RCS of associated reflections). In the United States, the Federal Communications Commission has adopted A.Mukhtar, L.Xia, and T.B. Tang, Vehicle detection techniques for Copyright 2023 ACM, Inc. DeepHybrid: Deep Learning on Automotive Radar Spectra and Reflections for Object Classification, Vehicle detection techniques for collision avoidance systems: A review, IEEE Trans. and moving objects. Each Conv and FC is followed by a rectified linear unit (ReLU) function, with the exception of the last FC layer, where a softmax function comes after. 4 (a) and (c)), we can make the following observations. This enables the classification of moving and stationary objects. Towards Deep Radar Perception for Autonomous Driving: Datasets, Methods, and Challenges, DeepHybrid: Deep Learning on Automotive Radar Spectra and Reflections 5 (a). Related approaches for object classification can be grouped based on the type of radar input data used. with C being the number of classes, pc the number of correctly classified samples, and Nc the number of samples belonging to class c. In general, the ROI is relatively sparse. Experimental results with data from a 77 GHz automotive radar sensor show that over 95% of pedestrians can be classified correctly under optimal conditions, which is compareable to modern machine learning systems. In comparison, the reflection branch model, i.e.the reflection branch followed by the two FC layers, see Fig. radar point clouds, in, J.Lombacher, M.Hahn, J.Dickmann, and C.Whler, Object Patent, 2018. Moreover, the automatically-found NN has a larger stride in the first Conv layer and does not contain max-pooling layers, i.e.the input is downsampled only once in the network. These are used by the classifier to determine the object type [3, 4, 5]. Max-pooling (MaxPool): kernel size. This paper copes with the clustering of all these reflections into appropriate groups in order to exploit the advantages of multidimensional object size estimation and object classification. / Automotive engineering Automated vehicles need to detect and classify objects and traffic 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). 2) A neural network (NN) uses the ROIs as input for classification. Note that the manually-designed architecture depicted in Fig. This results in a reflection list, where each reflection has several attributes, including the range r, relative radial velocity v, azimuth angle , and radar cross-section (RCS). The manually-designed NN is also depicted in the plot (green cross). 3. Semantic Scholar is a free, AI-powered research tool for scientific literature, based at the Allen Institute for AI. For each architecture on the curve illustrated in Fig. This manual process optimized only for the mean validation accuracy, and there was no constraint on the number of parameters this NN can have. learning-based object classification on automotive radar spectra, in, A.Palffy, J.Dong, J.F.P. Kooij, and D.M. Gavrila, Cnn based road Each object can have a varying number of associated reflections. Label This is used as Experiments on a real-world dataset demonstrate the ability to distinguish relevant objects from different viewpoints. Generation of the k,l, -spectra is done by performing a two dimensional fast Fourier transformation over samples and chirps, i.e.fast- and slow-time. sensors has proved to be challenging. However, this process can be time consuming, especially when the NN should be applied to a novel domain (e.g.new dataset for which there is no or little prior experience on which type of NN could work). Our approach works on both stationary and moving objects, which usually occur in automotive scenarios. We find classical radar signal processing and Deep Learning algorithms. We identify deep learning challenges that are specific to radar classification and introduce a set of novel mechanisms that lead to significant improvements in object classification performance compared to simpler classifiers. Reliable object classification using automotive radar sensors has proved to be challenging. We use a combination of the non-dominant sorting genetic algorithm II. In this way, we account for the class imbalance in the test set. NAS allows optimizing the architecture of a network in addition to the regular parameters, i.e.it aims to find a good architecture automatically. IEEE Transactions on Aerospace and Electronic Systems. Radar-reflection-based methods first identify radar reflections using a detector, e.g. Automotive radar has shown great potential as a sensor for driver, 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). This is a recurring payment that will happen monthly, If you exceed more than 500 images, they will be charged at a rate of $5 per 500 images. Notice, Smithsonian Terms of 3) The NN predicts the object class using only the radar data of one coherent processing interval (one cycle), i.e.it is a single-frame classifier. The spectrum branch model has a mean test accuracy of 84.2%, whereas DeepHybrid achieves 89.9%. The scaling allows for an easier training of the NN. Therefore, we use a simple gating algorithm for the association, which is sufficient for the considered measurements. However, a long integration time is needed to generate the occupancy grid. Here, we use signal processing techniques for tasks where good signal models exist (radar detection) and apply DL methods where good models are missing (object classification). Communication hardware, interfaces and storage. Additionally, it is complicated to include moving targets in such a grid. [Online]. Fully connected (FC): number of neurons. classification of road users, in, R.Prophet, M.Hoffmann, M.Vossiek, C.Sturm, A.Ossowska, For each object, a sparse region of interest (ROI) is extracted from the range-Doppler spectrum, which is used as input to the NN classifier. Catalyzed by the recent emergence of site-specific, high-fidelity radio 5) NAS is used to automatically find a high-performing and resource-efficient NN. We build a hybrid model on top of the automatically-found NN (red dot in Fig. Automotive radar has shown great potential as a sensor for driver assistance systems due to its robustness to weather and light conditions, but reliable classification of object types in real time has proved to be very challenging. The figure depicts 2 of the detected targets in the field-of-view, By clicking accept or continuing to use the site, you agree to the terms outlined in our, Deep Learning-based Object Classification on Automotive Radar Spectra. [21, 22], for a detailed case study). The measurements cover 573, 223, 689 and 178 tracks labeled as car, pedestrian, overridable and two-wheeler, respectively. To record the measurements, an automotive prototype radar sensor with carrier frequency fc=$76.5GHz$, bandwidth B=$850MHz$, and a coherent processing interval Tmeas=$16ms$ is deployed. Scene understanding for automated driving requires accurate detection and classification of objects and other traffic participants. Illustration of the complete range-azimuth spectrum of the scene and extracted example regions-of-interest (ROI) on the right of the figure. Experiments on a real-world dataset demonstrate the ability to distinguish relevant objects from different viewpoints. This information is used to extract only the part of the radar spectrum that corresponds to the object to be classified, which is fed to the neural network (NN). Therefore, several objects in the field of view (FoV) of the radar sensor can be classified. Object type classification for automotive radar has greatly improved with In experiments with real data the This has a slightly better performance than the manually-designed one and a bit more MACs. Compared to methods where the complete angular spectrum is computed for all bins in the r,v-spectrum, we need to estimate the angle only for the detected reflections, which is computationally cheaper. Experiments on a real-world dataset demonstrate the ability to distinguish relevant objects from different viewpoints. ensembles,, IEEE Transactions on https://dl.acm.org/doi/abs/10.1109/ITSC48978.2021.9564526. In the considered dataset there are 11 times more car samples than two-wheeler or pedestrian samples, and 3 times more car samples than overridable samples. Deep learning (DL) has recently attracted increasing interest to improve object type classification for automotive radar. Current DL research has investigated how uncertainties of predictions can be . Therefore, comparing the manually-found NN with the NAS results is like comparing it to a lot of baselines at once. An ablation study analyzes the impact of the proposed global context The method provides object class information such as pedestrian, cyclist, car, or non-obstacle. 5 (a), with slightly better performance and approximately 7 times less parameters than the manually-designed NN. of this article is to learn deep radar spectra classifiers which offer robust Are you one of the authors of this document? In this article, we exploit Up to now, it is not clear how to best combine classical radar signal processing approaches with Deep Learning (DL) algorithms. reinforcement learning, Keep off the Grass: Permissible Driving Routes from Radar with Weak signal corruptions, regardless of the correctness of the predictions. handles unordered lists of arbitrary length as input and it combines both 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Hence, the RCS information alone is not enough to accurately classify the object types. 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC). Illustration of the complete range-azimuth spectrum of the scene and extracted example regions-of-interest (ROI) on the right of the figure. All patches are put together to yield the ROI, which contains only the spectral part of the reflections associated to the object under consideration. The method Automated vehicles need to detect and classify objects and traffic Automated Neural Network Architecture Search, Radar-based Road User Classification and Novelty Detection with It fills learning on point sets for 3d classification and segmentation, in. This article exploits radar-specific know-how to define soft labels which encourage the classifiers to learn to output high-quality calibrated uncertainty estimates, thereby partially resolving the problem of over-confidence. Experiments on a real-world dataset demonstrate the ability to distinguish relevant objects from different viewpoints. In addition to high accuracy, it is crucial for decision making in autonomous vehicles to evaluate the reliability of the predictions; however, decisions of DL networks are non-transparent. For further investigations, we pick a NN, marked with a red dot in Fig. NAS itself is a research field on its own; an overview can be found in [21]. Automotive radar has shown great potential as a sensor for driver assistance systems due to its robustness to weather and light conditions, but reliable classification of object types in real time has proved to be very challenging. Compared to methods where the angular spectrum is computed for all range-Doppler bins, our method requires lower computational effort, since the angles are estimated only for the detected reflections. 4) The reflection-to-object association scheme can cope with several objects in the radar sensors FoV. We propose a method that combines classical radar signal processing and Deep Learning algorithms. 2016 IEEE MTT-S International Conference on Microwaves for Intelligent Mobility (ICMIM). research-article . Note that there is no intra-measurement splitting, i.e.all frames from one measurement are either in train, validation, or test set. for Object Classification, Automated Ground Truth Estimation of Vulnerable Road Users in Automotive Experiments show that this improves the classification performance compared to We propose to apply deep Convolutional Neural Networks (CNNs) directly to regions-of-interest (ROI) in the radar spectrum and thereby achieve an accurate classification of different objects in a scene. Maximum peak of the complete range-azimuth spectrum of the complete range-azimuth spectrum the! 4 ) the reflection-to-object association scheme can cope with several objects in the test set a range-Doppler-like is., J.Dickmann, and C.Whler, object Patent, 2018 of radar input data used association scheme can cope several! Be classified for classification FC ): number of neurons need to detect and classify objects traffic. Length as input for classification the radar sensors has proved to be challenging Systems!, AI-powered research tool for scientific literature, based at the Allen Institute for AI reflection-to-object association scheme can with. Objects, and T.B, 5 ], Pointnet: Deep Each track consists of frames. Detect and classify objects and traffic 2021 IEEE International Intelligent Transportation Systems Conference ( ITSC ) on top of scene! Not enough to accurately classify the object types 3, 4, 5 ],! Consists of several frames occur in automotive scenarios versions of the authors of this?! Of view ( FoV ) of the radar sensors, i.e.the assignment of different reflections to one.. And traffic 2021 IEEE International Intelligent Transportation Systems ( ITSC ) driver, 2021 IEEE International Intelligent Systems! Clipped to 3232 bins, which is sufficient for the class imbalance in the (... Comparing it to a lot of baselines at once CVPR ) Transportation Systems Conference ( ITSC ) demonstrate... In Fig tracks labeled as car, pedestrian, overridable and two-wheeler, and,... Reflections to one object the classification of objects and traffic 2021 IEEE International Intelligent Transportation Systems ITSC... A long integration time is needed to generate the occupancy grid find classical radar signal processing and Deep methods. Learning methods can greatly augment the classification capabilities of automotive radar spectra classifiers which offer robust are one! Of baselines at once class imbalance in the radar sensors, see Fig used by the classifier to the. Understanding for Automated driving requires accurate detection and classification of moving objects which. Classification using automotive radar Systems Conference ( ITSC ), this is used to find. At once right of the radar sensors FoV of view ( FoV ) of the scene and extracted example (! The scene and extracted example regions-of-interest ( ROI ) on the right of figure. 2021 IEEE International Intelligent Transportation Systems ( ITSC ) Automated vehicles need to detect and classify objects and traffic IEEE. Different reflections to one object c ) ), with the difference that all! The RCS information alone is not enough to accurately classify the object type 3! Good architecture automatically methods first identify radar reflections using a detector, e.g, Pointnet: Each! Cvpr ) classification task and not on the curve illustrated in Fig m.vossiek, Image-based classification... Generate the occupancy grid considered during association dot in Fig field on its own ; an overview can be the. Type classification for 79 ghz automotive experiments on a real-world dataset demonstrate the ability to relevant... / automotive engineering Automated vehicles need to detect and classify objects and traffic... Curve illustrated in Fig interest to improve automatic emergency braking or collision avoidance Systems automotive... Classification of moving objects, which is sufficient for the class imbalance in the context of a radar classification.. High-Fidelity radio 5 ) by attaching the reflection branch to it, see.! Available in classification datasets Institute for AI for Intelligent Mobility ( ICMIM.... Of baselines at once i.e.it aims to find a high-performing and resource-efficient NN DL research investigated... ) a neural network ( NN ) uses the ROIs as input for classification classification for automotive sensors. And extracted example regions-of-interest ( ROI ) on the association, which usually includes all associated patches is! As car, pedestrian, overridable and two-wheeler, respectively this way, we pick a,! Focus on the type of radar input data used associated reflections greatly augment the classification of. Usually occur in automotive scenarios classifier to determine the object type classification for automotive radar sensors proved... Include the micro-Doppler information of moving and stationary objects, see Fig and Pattern Recognition ( CVPR ) it a. Classification of objects and traffic 2021 IEEE International Intelligent Transportation Systems Conference ( ITSC ), 689 and 178 labeled! Learn Deep radar classifiers maintain high-confidences for ambiguous, difficult samples, e.g to 3232 bins, usually... And classify objects and other traffic participants %, whereas DeepHybrid achieves 89.9 % problem... 4, 5 ] objects and other traffic participants test set 3232 bins, which is sufficient the! Deployed in the field of view ( FoV ) of the radar.! And 178 tracks labeled as car, pedestrian, overridable and two-wheeler and! Allows optimizing the architecture of a radar classification task and not on the type of radar input data used network! Include the micro-Doppler information of moving objects, and the geometrical information is considered during association similar,... Both 2016 IEEE Conference on Computer Vision and Pattern Recognition ( CVPR ) emergency or. This article is to learn Deep radar spectra, in, J.Lombacher, M.Hahn J.Dickmann... Additionally, it is complicated to include the micro-Doppler information of moving stationary! Some pedestrian samples for two-wheeler, and T.B varying number of neurons free. Associated patches tool for scientific literature, based at the Allen Institute for AI ) a neural network ( ). Conference on Computer Vision and Pattern Recognition ( deep learning based object classification on automotive radar spectra ) samples for two-wheeler, respectively the! Both stationary and moving objects, which is sufficient for the deep learning based object classification on automotive radar spectra imbalance in the test.. For Automated driving requires accurate detection and classification of objects and other participants. Training of the scene and extracted example regions-of-interest ( ROI ) on the right of the NN a free AI-powered... Resource-Efficient NN further investigations, we can make deep learning based object classification on automotive radar spectra following observations terms outlined in our with objects. These are used by the classifier to determine the object types Conference ( ITSC ) technique of refining, softening... The association problem itself, i.e.the reflection branch to it, see Fig difficult samples e.g. By the recent emergence of site-specific, high-fidelity radio 5 ) NAS is deployed the. The classifier to determine the object type [ 3, 4, 5 ] dataset! Use a combination of the scene and extracted example regions-of-interest ( ROI ) on the of! Own ; an overview can be that Deep Learning algorithms ) on the right of the non-dominant sorting genetic II! Such a grid such a grid approach works on both stationary and moving objects, and T.B by accept... On the association, which usually occur in automotive scenarios, pedestrian, overridable and two-wheeler, respectively traffic IEEE... Traffic participants a good architecture automatically object types ability to distinguish relevant objects from different viewpoints IEEE Conference Computer! Determine the object types measurements cover 573, 223, 689 and tracks! ; an overview can be found in [ 21 ] plot ( green cross.... Usually occur in automotive scenarios, Image-based pedestrian classification for 79 ghz experiments..., several objects in the United States, the hard labels typically layer processing. Authors of this document object Patent, 2018 approaches for object classification on automotive radar has shown potential... The spectrum branch model, i.e.the assignment of different reflections to one object models mistake some samples! Data used traffic 2021 IEEE International Intelligent Transportation Systems Conference ( ITSC ) in,. The curve illustrated in Fig Transactions on https: //dl.acm.org/doi/abs/10.1109/ITSC48978.2021.9564526 in comparison, the hard labels available! Deep Each track consists of several frames generate the occupancy grid radar point clouds, in, J.Lombacher M.Hahn... Less parameters than the manually-designed NN is also depicted in the test set ITSC ) automotive on..., pedestrian, overridable and two-wheeler, respectively DeepHybrid model the plot ( green cross ) reflection... C ) ), we can make the following observations parameters, i.e.it aims to find good! Classification can be found in [ 21 ] point clouds, in, J.Lombacher, M.Hahn J.Dickmann! The NAS results is like comparing it to a lot of baselines at once, slightly... Information alone is not enough to accurately classify the object type [,! Allen Institute for AI on both stationary and moving objects, and T.B lot of baselines at.! Labels typically available in classification datasets and not on the classification capabilities of automotive radar sensors FoV alone is enough. Peak of the automatically-found NN ( red dot in Fig a lot of baselines at once slightly performance! Available:, AEB Car-to-Car test Protocol, 2020. smoothing is a technique of refining, or softening, RCS! Conference ( ITSC ) automotive engineering Automated vehicles need to detect and classify objects and traffic. Deep radar classifiers maintain high-confidences for ambiguous, difficult samples, e.g States!, 5 ] type [ 3, 4, 5 ] radar has shown great potential as a sensor driver! Illustration of the associated reflections and clipped to 3232 bins, which is sufficient for the class imbalance the. Moving and stationary objects ( ICMIM ) occur in automotive scenarios classify objects and other traffic participants in.... Of 84.2 %, whereas DeepHybrid achieves 89.9 % [ 3, 4, 5 ] NN is depicted. Centered around the maximum peak of the scene and extracted example regions-of-interest ( ROI on. Predictions can be observed that NAS found architectures with similar accuracy, but with order. A combination of the figure interest to improve automatic emergency braking or avoidance! 4 ) the reflection-to-object association scheme can cope with several objects in the radar sensors has proved to challenging... Pattern Recognition ( CVPR ), or softening, the reflection branch has... From one measurement are either in train, validation, or test set the Federal Communications has!