Adversarial Attacks on Traffic Sign Classification

Repository of the paper Evaluating Adversarial Attacks on Traffic Sign Classifiers beyond Standard Baselines at IEEE ICMLA 2024.

We decouple model architectures of LISA-CNN or GTSRB-CNN from the datasets and compare them to generic models for image classification (ResNet18, EfficientNet-B0, DenseNet-121, MobileNetv2, and ShuffleNetv2). Furthermore, we compare two attack settings, inconspicuous and visible, which are usually regarded without direct comparison. Our results show that standard baselines like LISA-CNN or GTSRB-CNN are significantly more susceptible than the generic ones.

Datasets

Download the GTSRB and LISA datasets to the dataset/ folder.

We use the predefined splits:

• LISA: 6834 images in total, 5467 train and 1367 test (80:20 split)
• GTSRB: 51839 images in total, 39209 train and 12630 test (75.64:24.36 split)

We used the pkl files for train and test from Zhong et al., available here.

Models

Three architectures, deliberately developed for the traffic sign classification task:

• CNN$_{small}$ is the original LISA-CNN as proposed by Eykholt et al.. We used the PyTorch implementation by Zhong et al. (code avalaible here) and extended it to the GTSRB data.

• CNN$_{large}$ is the original GTSRB-CNN based on the multi-scale CNN by Sermanet et al. and a later implementation by Yadav. We adapted the implementation from Zhong et al. and extended it to the LISA dataset.

• CNN-STN: we used the original implementation by Garcia et al. and extended it to the LISA dataset.

and five generic image classification models with a comparable number of parameters:

• ResNet18
• EfficientNet-B0
• DenseNet-121
• MobileNetv2
• ShuffleNetv2 with 1.0x output

Use notebooks 01_Train__Test_GTSRB_Models.ipynb and 01_Train__Test_GTSRB_Models.ipynb to train and evaluate models on the corresponding datasets.

Model Performance on Clean Data

Model	Accuracy on LISA, %	Accuracy on GTSDB, %	Number of parameters
CNN$_{small}$ (LISA-CNN)	99.71	98.13	0.73 M
CNN$_{large}$ (GTSRB-CNN)	99.78	98.91	16.54 M
CNN-STN	99.85	99.43	0.85 M
ResNet18	99.85	99.18	11.18 M
EfficientNet-B0	99.71	98.36	3.50 M
DenseNet-121	99.63	98.09	7.98 M
MobileNetv2	99.27	96.06	2.28 M
ShuffleNetv2	99.34	98.73	5.29 M

Adversarial Attacks

Use notebooks 03_Attack_GTSRB.ipynb and 04_Attack_LISA.ipynb to train and evaluate invonspicuous and visible attacks on the corresponding datasets.

Citation

If you find this code useful for your research, please cite our paper:

@InProceedings{pavlitska2024evaluating,
  author    = {Pavlitska, Svetlana and Müller, Leopold and Zöllner, J. Marius},
  title={Evaluating Adversarial Attacks on Traffic Sign Classifiers beyond Standard Baselines},
  booktitle = { International Conference on Machine Learning and Applications (ICMLA)},
  year      = {2024}
}