Home > Archive > 2024 > Volume 14 Number 4 (2024) >
IJML 2024 Vol.14(5): 119-128
DOI: 10.18178/ijml.2024.14.5.1468

Optimizing the Topology of Transformer Networks Using Modified Clonal Selection Algorithm: A Bio-Inspired Immunocomputing Approach

Ashish Kharel and Devinder Kaur
Electrical Engineering and Computer Science Department, University of Toledo, OH 43607 USA
Email: akharel@rockets.utoledo.edu (A.K.); dkaur@rockets.utoledo.edu (D.K.)
*Corresponding author

Manuscript received April 7, 2024; revised June 18, 2024; accepted June 28, 2024; published November 26, 2024

Abstract—This paper proposes the optimization of the Transformer model for analysis of sequential data using a modified clonal selection algorithm (mCSA). Transformers demonstrate better performance over Long Short-Term Memory (LSTM) deep networks when the input sequence is exceptionally long. They are good at capturing long-term dependencies in comparison to LSTM networks. However, this comparison is valid only if the hyperparameters are optimized correctly. Also, transformers are very sensitive to their hyperparameters. Designing the architecture of the transformer model for better performance is very complex and time-consuming. There have been other efforts using Bayesian, Grid Search, Blackbox, and metaheuristic optimization techniques for the optimization of the architecture of deep learning models. mCSA is a nature-inspired immunocomputing approach. The performance of the optimized transformer model has been compared with an unoptimized transformer model, genetic algorithm optimized transformer, Clonal Selection Algorithm optimized LSTM(CSA_LSTM), Clonal Selection Optimized Hybrid Convolutional Neural Network and LSTM network (CSA-CNN-LSTM, and Random Forest search algorithm. CSA optimized transformer model has consistently shown better performance in comparison to all other models for a variety of datasets such as IMDB-movie, SMS-Spam, and US Twitter Airline datasets. Here we also show that improper optimization of transformer hyperparameters can lead to degraded performance that cannot surpass even traditional ML approaches like random forest. We have also carried out ablation studies to understand the impact of various hyperparameters on the performance of our model.

Keywords—deep learning, clonal selection algorithm, immunocomputing, genetic algorithm, hyperparameters optimization, nature inspired algorithm, topology optimization, transformers

[PDF]

Cite: Ashish Kharel and Devinder Kaur, "Optimizing the Topology of Transformer Networks Using Modified Clonal Selection Algorithm: A Bio-Inspired Immunocomputing Approach," International Journal of Machine Learning vol. 14, no. 5, pp. 119-128, 2024.

Copyright © 2024 by the authors. This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).

General Information

  • E-ISSN: 2972-368X
  • Abbreviated Title: Int. J. Mach. Learn.
  • Frequency: Quarterly
  • DOI: 10.18178/IJML
  • Editor-in-Chief: Dr. Lin Huang
  • Executive Editor:  Ms. Cherry L. Chen
  • Abstracing/Indexing: Inspec (IET), Google Scholar, Crossref, ProQuest, Electronic Journals LibraryCNKI.
  • E-mail: ijml@ejournal.net
  • APC: 500USD


Article Metrics in Dimensions