INTRODUCTION TO _ NEURAL : NETWORKS _-% TF ay _ USING a= -X ee } ) | if : S N SIVANANDAM - S SUMATHI - S N DEEPAInformation contained in this work has been obtained by Tata McGraw-Hill, from sources believed to be reliable. However, neither Tata McGraw-Hill nor its authors guarantee the accuracy or completeness of any information including the program listings, published herein, and neither Tata McGraw-Hill nor its authors shall be responsible for any errors, omissions, or damages arising out of use of this information. This work is published with the understanding that Tata McGraw-Hill and its authors are supplying information but are not attempting to render engineering or other professional services. If such services are required, the assistance of an appropriate professional should be sought. @ Tata McGraw-Hill Copyright © 2006, by Tata McGraw-Hill Publishing Company Limited Second reprint 2006 RZLYCDRKRQLYC No part of this publication may be reproduced or distributed in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise or stored in a database or retrieval system without the prior written permission of the publishers. The program listings (if any) may be entered, stored and executed in a computer system, but they may not be reproduced for publication. ‘This edition can be exported from India only by the publishers, ‘Tata McGraw-Hill Publishing Company Limited. ISBN 0-07-0591 12-1 Published by the Tata McGraw-Hill Publishing Company Limited, 7 West Patel Nagar, New Delhi 110 008, typeset in Times at Script Makers, 19, Al-B, DDA Market, Pashchim Vihar, New Delhi 110 063 and Printed at S.P. Printers, E-120, Sector-7, Noida Cover: Shree Ram Enterprises Cover Design: Kapil Gupta, DelhiContents ai ay 2.2.1 Network Architecture _19 2.7.2 Setting the Weights 20 2.8 Artificial Neural Network (ANN) Terminologies _23 2.8.1 Weights 23 2.8.3 Sigmoidal Functions 24 2.8.4 Calculation of Net Input Using Matrix Multiplication Method 25 28.5 Bias 25 28.6 Threshold 26 2.9 Summary of Notations Used _27 ‘Summary 29 Review Questions 29x Contents 3.3.1. Hebbian Learning Rule _43 3.3.2 Perceptron Learning Rule 44 3.3.3 Delta Learning Rule (Widrow-Hoff Rule or Least Mean Square (LMS) Rule) 44 3.3.4 Competitive Learning Rule 47 3.3.5_Out Star Learning Rule 48 3.3.6 Boltzmann Learning 48 3.3.7 Memory Based Learning 48 3.4__Hebb Net 49 3.4.1 Architecture 49 3.4.2 Algorithm 50 3.4.3 Linear ility 50 Summary _ 56 Review Questions _57 Exemise Problems 57 4. Perceptron Networks 60 4.1 Introduction 60 42 Single Layer Perceptron_6/ 421 Architecture 61 4.2.2 Algorithm 62 4.2.3 Application Procedure 63 4.2.4 Perception Algorithm for Several Output Classes _63 4.3. Brief Introduction to Multilayer Perceptron Networks 84 Summary 85 Review Questions 85 Exercise Problems 85 5. Adaline and Madaline Networks 87 5.1 Introduction 87 5.2 Adaline 88 5.2.1 Architecture 88 5.2.2 Algorithm 88 5.2.3 Application Algorithm 89 5.3 Madaline 99 5.3.1 Architecture 99 5.3.2 MRI Algorithm /00 5.3.3 MRII Algorithm 101 Summary 106 Review Questions 107 Exercise Problems 107 6. Associative Memory Networks 109 6.1 Introduction 109 6.2 Algorithms for Pattern Association 1/0 6.2.1 Hebb Rule for Pattern Association 1/0 6.2.2 Delta Rule for Pattern Association 11]Contents xi 6.2.3 Extended Delta Rule 1 6.3 Hetero Associative Memory Neural Networks 1/3 6.3.1 Architecture //3 63.2 Application Algorithm 113 6.4 Auto Associative Memory Network 129 64.1 Architecture 129 64.2 Training Algorithm 129 6.4.3 Application Algorithm 130 6.4.4 Iterative Auto Associative Net 130 6.5 Bi-directional Associative Memory 150 6.5.1 Architecture 150 6.5.2 Types of Bi-directional Associative Memory Net 151 65.3 Application Algorithm 153 6.5.4 Hamming Distance 154 Summary 162 Review Questions 162 Exercise Problems 163 7. Feedback Networks 166 7.1 Introduction 166 7.2 Discrete Hopfield Net /67 7.2.1 Architecture 167 7.22 Training Algorithm 167 7.2.3 Application Algorithm 168 7.24 Analysis 168 7.3 Continuous Hopfield Net 179 7.4 Relation Between BAM and Hopfield Nets 187 Summary 181 Review Questions 18] Exercise Problems 18] ‘8. Feed Forward Networks 2.0°|\|\}\|]|]|]|1010 ag 8.1 Introduction _J84 8.2 __ Back Propagation Network (BPN) /85 8.2.1 Generalized Delta ing Rule (or) Back fion Rule 185 8.2.2 Architecture 186 8.2.3 Training Algorithm 187 8.24 Selection of Parameters 1/89 8.2.5 Learning in Back Propagation 190 8.2.6 Application Algorithm 192 8.2.7 Local Minima and Global Minima 192 8.2.8 Merits and Demerits of Back Propagation Network _193 8.2.9 Applications 193 8.3 Radial Basis Function Network (RBFN) 2/2 8.3.1 Architecture 2/3xii Contents 8.3.2. Training Algorithm for an RBFN with Fixed Centers 2/3 Summary 217 Review Questions 218 Exercise Problems 218 9. Self Organizing Feature Map 9.1 Introduction 220 9.2 _ Methods Used for Determining the Winner 22] 9.3 Kohonen Self Organizing Feature Maps (SOM) 22! 93.1 Architecture 222 9.3.2 Training Algorithm 223 9.4 Learning Vector Quantization (LVQ) 237 9.4.1 Architecture 238 9.4.2 Training Algorithm 238 9.4.3 Variants of LVQ 239 9.5 Max Net 245 9.5.1 Architecture 246 9.5.2 Application Procedure 246 9.6 Mexican Hat 249 9.6.1 Architecture 249 9.6.2 Training Algorithm 250 9.7 Hamming Net 253 9.7.1 Architecture 254 9.7.2 Application Procedure 254 Summary 257 Review Questions 257 Exercise Problems 258 10. Counter Propagation Network 10.1 Introduction 260 10.2 Full Counter Propagation Network (Full CPN) 267 10.2.1 Architecture 26/ 10.2.2 Training Phases of Full CPN 263 10.2.3 Training Algorithm 264 10.2.4 Application Procedure 265 10.3 Forward only Counter Propagation Network 270 10.3.1 Architcture 270 10.3.2 Training Algorithm 271 10.3.3 Application procedure 272 Summary 274 Review Questions 274 Exercise Problems 274 11. Adaptive Resonance Theory 11.1 Introduction 277 11.2 ART Fundamentals 2783 114 Contents xiii 11.2.1 Basic Architecture 279 11.2.2 Basic Operation 279 11.2.3 Learning in ART 280 11.2.4 Basic Training Steps 280 ART 1 280 11.3.1 Architecture 287 11.32 Algorithm 282 ART 2 299 11.4.1 Architecture 299 11.4.2 Training Algorithm 300 Summary 309 Review Questions 309 Exercise Problems 310 12. Special Networks Bo 121 122 123 124 125 12.6 12.7 128 129 Introduction 312 Probabilistic Neural Network 3/2 12.2.1 Architecture 312 12.2.2 Training Algorithm 313 12.2.3 Application Algorithm 3/4 Cognitron 314 123.1 Architecture 314 12.3.2 Training 31/5 12.3.3 Excitatory Neuron 3/5 12.34 Inhibitory Neuron 316 12.3.5 Training Problems 3/7 Neocognitron 318 12.4.1 Architecture 3/8 12.4.2 S-cells: Simple Cells 3/9 12.4.3 C-cells: Complex Cells 319 12.4.4 Algorithm Calculations 319 12.4.5 Training 320 Boltzmann Machine 320 125.1 Architecture 320 12.5.2 Application Algorithm 321 Boltzmann Machine with Learning 322 12.6.1 Architecture 323 12.6.2 Algorithm 323 Gaussian Machine 325 Cauchy Machine 326 Optical Neural Networks 327 12.9.1 Electro-optical Matrix Multipliers 327 12.9.2 Holographic Correlators 328 12.10 Simulated Annealing 329 12.10.1 Algorithm 329121 12.12 12.13 12.14 12.15 12.10.2 Structure of Simulated Annealing Algorithm 331 12.10.3 When to Use Simulated Annealing 332 Cascade Correlation 333 12.111 Architecture 333 12.11.2 Steps Involved in Forming a Cascade Correlation Network 334 12.11.3 Training Algorithm 334 Spatio-temporal Neural Network 336 12.12.1 Input Dimensions 336 12.12.2 Long- and Short-term Memory in Spatio-temporal Connectionist Networks 337 12.12.3 Output, Teaching, and Error 337 12.12.4 Taxonomy for Spatio-temporal Connectionist Networks 338 12.12.5 Computing the State Vector 339 12.12.6 Computing the Output Vector 339 12.12.7 Initializing the Parameters 339 12.12.8 Updating the Parameters 340 ‘Support Vector Machines 340 12.13.1 Need for SVMs 342 12.13.2 Support Vector Machine Classifiers 343 Pulsed Neural Networks 345 12.14.1 Pulsed Neuron Model (PN Model) 345 Neuro-dynamic Programming 347 12.15.1 Example of Dynamic Programming 349 12.15.2 Applications of Neuro-dynamic Programming 350 Summary 350 Review Questions 350 13. Applications of Neural Networks 352 13.1 13.2 13.3 13.4 13.5 13.6 Applications of Neural Networks in Arts 353 13.1.1 Neural Networks 353 13.1.2 Applications 355 13.1.3 Conclusion 357 Applications of Neural Networks in Bioinformatics 358 13.2.1 A Bioinformatics Application for Neural Networks 358 Use of Neural Networks in Knowledge Extraction 360 13.3.1 Artificial Neural Networks on Transputers 36/ 13.3.2. Knowledge Extraction from Neural Networks 363 Neural Networks in Forecasting 367 13.4.1 Operation of a Neural Network 368 13.4.2 Advantages and Disadvantages of Neural Networks 368 13.4.3 Applications in Business 369 Neural Networks Applications in Bankruptcy Forecasting 371 13.5.1 Observing Data and Variables 372 13.5.2 Neural Architechture 372 13.5.3 Conclusion 374 Neural Networks in Healthcare 37413.7 138 Contents | xv 13.6.1 Clinical Diagnosis 374 13.6.2 Image Analysis and Interpretation 376 13.6.3 Signal Analysis and Interpretation 377 13.6.4 Drug Development 378 Application of Neural Networks to Intrusion Detection 378 13.7.1 Classification of Intrusion Detection Systems 378 13.7.2 Commercially Available Tools 378 13.7.3. Application of Neural Networks to Intrusion Detection 380 13.7.4 DARPA Intrusion Detection Database 380 13.7.5 Georgia University Neural Network IDS 380 13.7.6 MIT Research in Neural Network IDS 387 13.7.7 UBILAB Laboratory 38/ 13.7.8 Research of RST Corporation 387 13.7.9 Conclusion 382 Neural Networks in Communication 382 13.8.1 Using Intelligent Systems 383 13.8.2 Application Areas of Artificial Neural Networks 383 13.8.3. European Initiatives in the Field of Neural Networks 385 13.8.4 Application of Neural Networks in Efficient Design of RF and Wireless Circuits 386 13.8.5 Neural Network Models of Non-linear Sub-systems 387 13.8.6 Modeling the Passive Elements 388 13.8.7 Conclusion 389 13.9 Neural Networks in Robotics 389 13.9.1, Natural Landmark Recognition using Neural Networks for Autonomous Vacuuming Robots 389 13.9.2 Conclusions 395 13.10 Neural Network in Image Processing and Compression 395 13.11 13.10.1 Windows based Neural Network Image Compression and Restoration 395 13.10.2. Application of Artificial Neural Networks for Real Time Data Compression 401 13.10.3 Image Compression using Direct Solution Method Based Neural Network 406 13.104 Application of Neural Networks to Wavelet Filter Selection in Multispectral Image Compression 41} 13.10.5 Human Face Detection in Visual Scenes 413 13.10.6 Rotation Invariant Neural Network-based Face Detection 416 Neural Networks in Business 42/ 13.11.1 Neural Network Applications in Stock Market Predictions—A Methodology Analysis 427 13.11.2 Search and Classification of “Interesting” Business Applications in the World Wide Web Using a Neural Network Approach 427 13.12, Neural Networks in Control 432 13.13 13.12.1 Basic Concept of Control Systems 432 13.12.2 Applications of Neural Network in Control Systems 433 Neural Networks in Pattern Recognition 440xvi Contents 13.13.1 Handwritten Character Recognition 440 13.14 Hardware Implementation of Neural Networks 448 13.14.1 Hardware Requirements of Neuro-Computing 448 13.14.2 Electronic Circuits for Neurocomputing 450 1.14.3 Weights Adjustment using Integrated Circuit 452 14. Ay of Special Networks 455 14.1 Temporal Updating Scheme for Probabilistic Neural Network with Application to Satellite Cloud Classification 455 14.1.1 Temporal Updating for Cloud Classification 457 14.2 Application of Knowledge-based Cascade-correlation to Vowel Recognition 458 14.2.1 Description of KBCC 459 14.2.2 Demonstration of KBCC: Peterson-Barney Vowel Recognition 460 14.2.3 Discussion 463 14,3 Rate-coded Restricted Boltzmann Machines for Face Recognition 464 14.3.1 Applying RBMs to Face Recognition 465 14.3.2 Comparative Results 467 14.3.3. Receptive Fields Learned by RBMrate 469 14.3.4 Conclusion 469 144 MPSA: A Methodology to Parallelize Simulated Annealing and its Application to the Traveling Salesman Problem 470 14.4.1 Simulated Annealing Algorithm and the Traveling Salesman Problem 470 14.4.2 Parallel Simulated Annealing Algorithms 47/ 14.4.3 Methodology to Paralletize Simulated Annealing 471 14.44 TSP-Parallel SA Algorithm Implementation 474 144.5 TSP-Parallel SA Algorithm Test. 474 14.5 Application of “Neocognitron” Neural Network for Integral Chip Images Processing 476 14.6 Generic Pretreatment for Spiking Neuron Application on Lip Reading with STANN (Spatio-Temporal Artificial Neural Networks) 480 14.6.1 STANN 480 14.6.2 General Classification System with STANN 48] 14.6.3 A Generic Pretreatment 482 14.6.4 Results 483 14.7 Optical Neurat Networks in Image Recognition 484 14.7.1 Optical MVM 485 14.7.2 Input Test Patterns 485 14.7.3 Mapping TV Gray-levels to Weights 486 14.7.4 Recall in the Optical MVM 487 14.7.5 LCLV Spatial Characteristics 488 14.7.6 Thresholded Recall 489 14.7.7 Discussion of Results 489 15, Neural Network Projects with MATLAB. 491 15.1 Brain Maker to Improve Hospital Treatment using ADALINE 491 15.1.1 Symptoms of the Patient 49215.2 15.3 15.4 15.5 15.1.2 Need for Estimation of Stay 493 15.1.3 ADALINE 493 15.1.4 Problem Description 493 15.1.5 Digital Conversion 493 15.1.6 Data Sets 494 15.1.7 Sample Data 494 15.1.8 Program for ADALINE Network 495 15.1.9 Program for Digitising Analog Data 498 15.1.10 Program for Digitising the Target 498 15.1.1 Program for Testing the Data 499 15.1.12 Simulation and Results 499 15.1.13 Conclusion S01 Breast Cancer Detection Using ART Network 502 15.2.1 Art 1 Classification Operation 502 15.2.2 Data Representation Schemes 503 15.2.3. Program for Data Classification using Art Network 508 15.2.4 Simulation and Results 5/7 15.2.5 Conclusion 512 Access Control by Face Recognition using Backpropagation Neural Network 513 15.3.1 Approach 513 15.3.2 Face Training and Testing Images 5/5 15.3.3 Data Description 516 15.3.4 Program for Discrete Training Inputs 518 15.3.5 Program for Discrete Testing Inputs 521 15.3.6 Program for Continuous Training Inputs 523 15.3.7 Program for Continuous Testing Inputs 527 15.3.7 Simulation 529 15.3.8 Results 530 15.3.9 Conclusion 531 Character Recognition using Kohonen Network 531 15.4.1 Kohonen’s Learning Law 53! 15.4.2 Winner-take-all 532 15.4.3 Kohonen Self-organizing Maps 532 15.4.4 Data Representation Schemes 533 15.4.5 Description of Data 533 15.4.6 Sample Data 534 15.4.7 Kohonen’s Program 536 15.4.8 Simulation Results 540 15.4.9 Kohonen Results 540 154.10 Observation 540 15.4.1 Conclusion 540 Classification of Heart Disease Database using Learning Vector Quantization Artificial Neural Network 541 15.5.1 Vector Quantization 54]xviii Conients 15.5.2 Learning Vector Quantization 541 15.5.3 Data Representation Scheme 542 15.5.4 Sample of Heart Disease Data Sets 543 15.5.5 LVQ Program $43 15.5.6 Input Format 550 15.5.7 Output Format 557 15.5.8 Simulation Results 557 15.5.9 Observation 552 155.10 Conclusion 552 15.6 Data Compression using Backpropagation Network 552 15.6.1 Back Propagation Network 552 15.6.2 Data Compression 553 15.6.3 Conventional Methods of Data Compression 553 15.6.4 Data Representation Schemes 554 15.6.5 Sample Data 555 15.6.6 Program for Bipolar Coding 556 15.6.7 Program for Implementation of Backpropagation Network for Data Compression 557 15.6.8 Program for Testing 561 15.6.9 Results 563 156.10 Conclusion 565 15.7 System Identification using CMAC 565 15.7.1 Overview of System Identification 566 15.7.2 Applications of System Identification 566 15.7.3. Need for System Identification 568 15.7.4 Identification Schemes 569 15.7.5 Least Squares Method for Self Tuning Regulators 569 15.7.6 Neural Networks in System Identification 570 15.7.7 Cerebellar Model Arithmetic Computer (CMAC) 572 15.7.8 Properties of CMAC 575 15.7.9 Design Criteria 576 15.7.10 Advantages and Disadvantages of CMAC 578 15.7.1 Algorithms for CMAC 579 Program 58/ Results 596 15.7.14 Conclusion 599 15.8 Neuro-fuzzy Control Based on the Nefcon-model under MATLAB/SIMULINK 600 15.8.1 Learning Algorithms 60/ 15.8.2 Optimization of a Rule Base 602 15.8.3 Description of System Error 602 15.8.4 Example 604 15.8.5 Conclusion 60716. Fuzzy Systems 16.1 Introduction 608 16.2 History of the Development of Fuzzy Logic 608 16.3 Operation of Fuzzy Logic 609 16.4 Fuzzy Sets and Traditional Sets 609 16.5 Membership Functions 6/0 16.6 Fuzzy Techniques 610 16.7 Applications 612 16.8 Introduction to Neuro Fuzzy Systems 6/2 16.8.1 Fuzzy Neural Hybrids 614 16.8.2 Neuro Fuzzy Hybrids 616 Summary 619 Appendix—MATLAB Neural Network Toolbox Al A Simple Example 621 A.LI Prior to Training 622 A.L.2 Training Error 622 1.3 Neural Network Output Versus the Targets 623 A.2_ A Neural Network to Model sin(x) 623 A2.1 Prior to Training 624 A22 Training Error 624 A.23 Neural Network Output Versus the Targets 625 3 Saving Neural Objects in MATLAB 626 A3.1 Examples 63] A4 Neural Network Object: 637 A4.1 Example 634 AS. Supported Training and Learning Functions 637 A.S.1 Supported Training Functions 638 AS.2 Supported Learning Functions 638 A.5.3 Transfer Functions 638 A.5.4 Transfer Derivative Functions 639 ASS i and Bias Initialization Functions 639 ASO t Derivative Functions. 639 A6 Neural Network Toolbox GUI 639 A6.1 Introduction to GUI 640 Summary 645 Contentsa You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.a You have either reached 2 page thts unevalale fer vowing or reached your ievina tit for his book.Acknowledgements First of all, the authors would like to thank the Almighty for granting them perseverance and achievements. Dr SN Sivanandam, Dr $ Sumathi and § N Deepa wish to thank Mr V Rajan, Managing Trustee, PSG Institutions, Mr C R Swaminathan, Chief Executive, and Dr S Vijayarangan, Principal, PSG College of Technology, Coimbatore, for their whole-hearted cooperation and encouragement provided for this endeavor. The authors are grateful for the support received from the staff members of the Electrical and Electronics Engineering and Computer Science and Engineering departments of their college. Dr Sumathi owes much to her daughter $ Priyanka, who cooperated even when her time was being monopolized with book work. She feels happy and proud for the steel-frame support rendered by her husband. She would like to extend whole-hearted thanks to her parents and parents-in-law for their constant support. She is thankful to her brother who has always been the “Stimulator” for her progress. Mrs S N Deepa wishes to thank her husband Mr T $ Anand, her daughter Nivethitha T $ Anand and her family for the support provided by them. Thanks are also due to the editorial and production teams at Tata McGraw-Hill Publishing Company Limited for their efforts in bringing out this book.Scope of neural networks and DPoaHvrpmo * How neural network is used to learn patterns and relationships in data. # The aim of neural networks. | # About fuzzy logic . Use of MATLAB to develop the applications based on neural networks. bia Introduction to Neural Networks Artificial neural networks are the result of academic investigations that use mathematical formulations to model nervous system operations. The resulting techniques are being successfully applied in a variety of everyday business applications. Neural networks (NNs) represent a meaningfully different approach to using computers in the work- place. A neural network is used to learn patterns and relationships in data. The data may be the results of2 Introduction to Neural Networks a market research effort, a production process given varying operational conditions, or the decisions of a loan officer given a set of loan applications. Regardless of the specifics involved, applying a neural network is substantially different from traditional approaches. Traditionally a programmer or an analyst specifically ‘codes’ for every facet of the problem for the computer to ‘understand’ the situation. Neural networks do not require explicit coding of the problems. For example, to generate a model that performs a sales forecast, a neural network needs to be given only raw data related to the problem. The raw data might consist of history of past sales, prices, competitors’ prices and other economic variables. The neural network sorts through this information and produces an understanding of the factors impacting sales. The model can then be called upon to provide a prediction of future sales given a forecast of the key factors. These advancements are due to the creation of neural network learning rules, which are the algo- rithms used to ‘learn’ the relationships in the data, The learning rules enable the network to ‘gain know!- edge’ from available data and apply that knowledge to assist a manager in making key decisions, What are the Capabilities of Neural Networks? In principle, NNs can compute any computable function, i.e. they can do everything a normal digital computer can do. Especially anything that can be represented as a mapping between vector spaces can be approximated to arbitrary precision by feedforward NNs (which is the most often used type). In practice, NNs are especially useful for mapping problems, which are tolerant of some errors, have lots of example data available, but to which hard and fast rules cannot easily be applied. However, NNs are, as of now, difficult to apply successfully to problems that concern manipulation of symbols and memory. Who is Concerned with Neural Networks? Neural Networks are of interest to quite a lot of people from different fields: ‘© Computer scientists want to find out about the properties of non-symbolic information processing with neural networks and about learning systems in general. * Engineers of many kinds want to exploit the capabilities of neural networks in many areas (e.g. signal processing) to solve their application problems. © Cognitive scientists view neural networks as a possible apparatus to describe models of thinking and.conscience (High-level brain function). © Neuro-physiologists use neural networks to describe and explore medium-level brain function (e.g. memory, sensory system). * Physicists use neural networks to model phenomena in statistical mechanics and for a lot of other tasks. * Biologists use Neural Networks to interpret nucleotide sequences. ‘* Philosophers and some other people may also be interested in Neural Networks to gain knowledge about the human systems namely behavior, conduct, character, intelligence, brilliance and other psychological feelings. Environmental nature and related functioning, marketing business as well as designing of any such systems can be implemented via Neural networks. The development of Artificial Neural Network started $0 years ago. Artificial neural networks (ANNs) are gross simplifications of real (biological) networks of neurons. The paradigm of neural networks,Introduction to Neural Networks 3 which began during the 1940s, promises to be a very important tool for studying the structure-function relationship of the human brain, Due to the complexity and.incomplete understanding of biological neurons, various architectures of artificial neural networks have been reported in the literature. Most of the ANN structures used commonly for many applications often consider the behavior of a single neu- ron as the basic computing unit for describing neural information processing operations. Each comput- ing unit, i. the artificial neuron in the neural network is based on the concept of an ideal neuron. An ideal neuron is assumed to respond optimally to the applied inputs. However, experimental studies in neuro-physiology show that the response of a biological neuron appears random and only by averaging many observations it is possible to obtain predictable results. Inspired by this observation, some re~ searchers have developed neural structures based on the concept of neural populations. In common with biological neural networks, ANNs can accommodate many inputs in parallel and encode the information in a distributed fashion. Typically the information that is stored in a neural net is shared by many of its processing units. This type of coding is in sharp contrast to traditional memory schemes, where a particular piece of information is stored in only one memory location. The recall process is time consuming and generalization is usually absent. The distributed storage scheme provides ‘many advantages, most important of them being the redundancy in information representation, Thus, an ANN can undergo partial destruction of its structure and still be able to function well. Although redun- dancy can also be built into other types of systems, ANN has a natural way of implementing this. The result is a natural fault-tolerant system which is very similar to biological systems, ‘The aim of neural networks is to mimic the human ability to adapt to changing circumstances and the current environment. This depends heavily on being able to learn from events that have happened in the past and to be able to apply this to future situations. For example the decisions made by doctors are rarely based on a single symptom because of the complexity of the human body; since one symptom could signify any number of problems. An experienced doctor is far more likely to make a sound deci- sion than a trainee, because from his past experience he knows what to look out for and what to ask, and may have etched on his mind a past mistake, which be will not repeat. Thus the senior doctor is in a superior position than the trainee. Similarly it would be beneficial if machines, too, could use past events as part of the criteria on which their decisions are based, and this is the role that artificial neural net- works seek to fill Artificial neural networks consist of many nodes, i.e. processing units analogous to neurons in the brain. Each node has a node function, associated with it which along with a set of local parameters determines the output of the node, given an input. Modifying the local parameters may alter the node function. Artificial Neural Networks thus is an information-processing system. In this information-pro- cessing system, the elements called neurons, process the information. The signals are transmitted by means of connection links. The links possess an associated weight, which is multiplied along with the incoming signal (net input) for any typical neural net. The output signal is ob- tained by applying activations to the net input. The neural net can generally be a single layer or a multi- layer net. The structure of the simple artificial neural net is shown in Fig. 1 Figure 1.1 shows a simple artificial neural net with two input neurons (x), Xz) and one output neuron (y). The inter- connected weights are given by w, and w5. In a single layer net there is a single layer of weighted interconnections, Fig. 1.4 | 4 Simple artical Neural Net 24 Introduction to Neural Networks A typical multi-layer artificial neural network, abbreviated as MNN, com-« prises an input layer, output layer and hidden (intermediate) layer of neurons. MNNs are often called layered net- works. They can implement arbitrary y complex input/output mappings or de- | RN, LD— yalty) cision surfaces separating different pat- terns. A three-layer MNN is shown in jg. 1.2, and a simpli diagram Teecueatitig atte MNKs Input Layer Hidden Layer Output Layer layer of input units is connected to a yi(t)) ~ yell) Sy layer of hidden units, which is connected {A Densely interconnected Three-layered Static to the layer of output units. The activity jpig/4/2)] Neural Network. Each Shaded Circle, or Node, of neurons in the input layer represents Hissssne bs eee cone the raw information that is fed into the network. The activity of neurons in the | mm hidden layer is determined by the activi- xy No +} N5 ties of the input neurons and the con- | | necting weights between the input and Input Layer Hidden Layer Output Layer hidden units. Similarly, the behavior of. the output units depends on the activity A Block Diagram Representation of a of the neurons in the hidden layer and | P'™9 | three-layered MNN the connecting weights between the hid- den and the output layers. This simple neural structure is interesting because neurons in the hidden layers are free to construct their own representation of the input. MNNS provide an increase in computational power over a single-layer neural network unless there is nonlinear activation function between layers. Many capabilities of neural networks, such as nonlinear functional approximation, learning, generalization, etc are in fact performed due to the nonlinear activa- tion function of each neuron. y, ANNs have become a technical folk legend. The market is flooded with new, increasingly technical software and hardware products, and many more are sure to come. Among the most popular hardware implementations are Hopfield, Multilayer Perceptron, Self-organizing Feature Map, Learning Vector Quantization, Radial Basis Function, Cellular Neural, and Adaptive Resonance Theory (ART) networks, Counter Propagation networks, Back Propagation networks, Neo-cognitron, etc. As a result of the exist- ence of all these networks, the application of the neural network is increasing tremendously. Thus artificial neural network represents the major extension to computation. They perform the op- erations similar to that of the human brain. Hence it is reasonable to expect a rapid increase in our understanding of artificial neural networks leading to improved network paradigms and a host of appli- cation opportunities ‘1.3. The Rise of Neurocomputing a A majority of information processing today is carried out by digital computers. This has led to the widely held misperception that information processing is dependent on digital computers. However, ifIntroduction to Neural Networks 5 we look at cybernetics and the other disciplines that form the basis of information science, we see that information processing originates with living creatures in their struggle to survive in their environ- ments, and that the information being processed by computers today accounts for only a small part — the automated portion — of this. Viewed in this light, we can begin to consider the possibility of infor- mation processing devices that differ from conventional computers. In fact, research aimed at realizing a variety of different types of information processing devices is already being carried out, albeit in the shadows of the major successes achieved in the realm of digital computers. One direction that this research is taking is toward the development of an information processing device that mimics the struc- tures and operating principles found in the information processing systems possessed by humans and other living creatures. Digital computers developed rapidly in and after the late 1940's and after originally being applied to the field of mathematical computations, have found expanded applications in a variety of areas, like text (word), symbol, image and voice processing, i.e. pattern information processing, robotic control and artificial intelligence. However, the fundamental structure of digital computers is based on the principle cof sequential (serial) processing, which has little if anything in common with the human nervous system. The human nervous system, it is now known, consists of an extremely large number of nerve cells, or neurons, which operate in parallel to process various types of information. By taking a hint from the structure of the human nervous system, we should be able to build a new type of advanced parallel information processing device. In addition to the increasingly large volumes of data that we must process as a result of recent devel- opments in sensor technology and the progress of information technology, there is also a growing re- quirement to simultaneously gather and process huge amounts of data from multiple sensors and other sources. This situation is creating a need in various fields to switch from conventional computers that process information sequentially, to parallel computers equipped with multiple processing elements aligned to operate in parallel to process information. Besides the social requirements just cited, a number of other factors have been at work during the 1980's to prompt research on new forms of information processing devices. For instance, recent neuro- physiological experiments have shed considerable light on the structure of the brain, and even in fields such as cognitive science, which study human information processing processes at the macro level, we are beginning to see proposals for models that call for multiple processing elements aligned to operate in parallel. Research in the fields of mathematical science and physics is also concentrating more on the mathematical analysis of systems comprising multiple elements that interact in complex ways. These factors gave birth to a major research trend aimed at clarifying the structures and operating principles inherent in the information processing systems of human beings and other animals, and constructing an information processing device based on these structures and operating principles. The term ‘neurocomputing’ is used to refer to the information engineering aspects of this research. Dr. Cleve Moler, chief scientist at MathWorks, Inc., originally wrote MATLAB to provide easy access to matrix software developed in the LINPACK and EISPACK projects. The first version was written in the late 1970s for use in courses in matrix theory, linear algebra, and numerical analysis. MATLAB is therefore built upon a foundation of sophisticated matrix software, in which the basic data element is a matrix that does not require predimensioning.6 Introduction to Neural Networks MATLAB is a product of The MathWorks, Inc. and is an advanced interactive software package specially designed for scientific and engineering computation. The MATLAB environment integrates graphical illustrations with precise numerical calculations, and is a powerful, easy-to-use, and compre- hensive tool for performing all kinds of computations and scientific data visualization. MATLAB has proven to be a very flexible and useful tool for solving problems in many areas. MATLAB is a high- performance language for technical computing. It integrates computation, visualization and program- ming in an easy-to-use environment where problems and solutions are expressed in familiar mathemati- cal notation, Typical areas of application of MATLAB include: ‘© Math and computation * Algorithm development ‘* Modeling, simulation and prototyping * Data analysis, exploration, and visualization © Scientific and engineering graphics * Application development, including graphical user interface building MATLAB is an interactive system whose basic element is an array that does not require dimensioning. This helps in solving many computing problems, especially those with matrix and vector formulations, in a fraction of the time it would take to write a program in a scalar non-interactive language such as C or FORTRAN. Mathematics is the common language of science and engineering. Matrices, differential equations, arrays of data, plots and graphs are the basic building blocks of both applied mathematics and MATLAB. It is the underlying mathematical base that makes MATLAB accessible and powerful. MATLAB allows expressing the entire algorithm in a few dozen lines, to compute the solution with great accuracy in about a second. Therefore it is especially helpful for technical analysis, algorithm prototyping and application development. MATLAB‘s two-and three-dimensional graphics are object oriented. MATLAB is thus both an envi- ronment and a matrix/vector-oriented programming language, which enables the user to build own reusable tools. The user can create his own customized functions and programs (known as M-files) in MATLAB code. The Toolbox is a specialized collection of M-files for working on particular classes of problems. MATLAB Documentation Set has been written, expanded and put online for ease of use. The set includes online help, as well as hypertext-based and printed manuals. The commands in MATLAB are expressed in a notation close to that used in mathematics and engineering. There is a very large set of these commands and functions, known as MATLAB M-files. As a result, solving problems through MATLAB is faster than the other traditional programming. It is easy to modify the funetions since most of the M-files can be opened and modified. For ensuring high performance, the MATLAB software has been written in optimized C and coded in assembly language. The main features of MATLAB can be summarized as: * Advance algorithms for high-performance numerical computations, especially in the field of ma- trix algebra. * A large collection of predefined mathematical functions and the ability to define one’s own func- tions, * Two- and three-dimensional graphics for plotting and displaying data. * A complete online help system. * Powerful, matrix/vector-oriented, high-level programming language for individual applications.Introduction to Neural Networks 7 * Ability to cooperate with programs written in other languages and for importing and exporting formatted data. * Toolboxes available for solving advanced problems in several application areas, Figure 1.4 shows the main features and capabilities of MATLAB. ‘MATLAB Programming Language User-written Functions Built-in Functions Graphics © 2-D Graphics + 3-D Graphics * Color and onan Toolboxes + Signal Processing ‘+ Image Processing + Control System ‘* Optimization ‘= Neural Networks * Communications + Robust Control GRR] Features and Capabilities of MATLAB An optional extension of the core of MATLAB called SIMULINK is also available. SIMULINK means SIMUlating and LINKing the environment. SIMULINK is an environment for simulating linear and non-linear dynamic systems, by constructing block diagram models wit user interface. an easy to use graphical8 Introduction to Neural Networks SIMULINK is a MATLAB toolbox designed for the dynamic simulation of linear and non-linear systems as well as continuous and discrete-time systems. It can also display information graphically. MATLAB is an interactive package for numerical analysis, matrix computation, control system design, and linear system analysis and design available on most CAEN (Computer Aided Engineering Network) platforms (Macintosh, PCs, Sun, and Hewlett-Packard). In addition to the standard functions provided by MATLAB, there exist a large set of Toolboxes, or collections of functions and procedures, available as part of the MATLAB package. Toolboxes are libraries of MATLAB functions used to customize MATLAB for solving particular class of problems. Toolboxes are a result of some of the world’s top researches in specialized fields. They are equivalent to prepackaged ‘off-the-shelf” software solution for a particular class of problem or technique. It is a collection of special files called M-files that extend the functionality of the base program. The various Toolboxes available are: * Control System: Provides several features for advanced control system design and analysis. + Communications: Provides functions to model the components of a communication system’s physical layer. * Signal Processing: Contains functions to design analog and digital filters and apply these filters to data and analyze the results, «System Identification: Provides features to build mathematical models of dynamical systems based on observed system data. * Robust Control: Allows users to create robust multivariable feedback control system designs based on the concept of the singular-value Bode plot. * Simulink: Atlows you to model dynamic systems graphically. * Neural Network: Allows you to simulate neural networks. * Fuzzy Logic: Allows for manipulation of fuzzy systems and membership functions. * Image Processing: Provides access to a wide variety of functions for reading, writing, and filtering images of various kinds in different ways. : Includes a wide variety of system analysis tools for varying matrices. jion: Contains basic tools for use in constrained and unconstrained minimization prob- # Spline: Can be used to find approximate functional representations of data sets. * Symbolic: Allows for symbolic (rather than purely numeric) manipulation of functions. © User Interface Utili includes tools for creating dialog boxes, menu utilities, and other user interactions for script files. MATLAB has been used as an efficient tool, all over this text to develop the applications based on Neural Nets. Review Questions 1.1 How did neurocomputing originate? 1.2 What is a multilayer net? Describe with a neat sketch.Introduction to Neural Networks 1.3. State some of the popular neural networks. 1.4 Briefly discuss the key characteristics of MATLAB. 1.5. List the basic arithmetic operations that can be performed in MATLAB. 1.6 What is the necessity of SIMULINK package available in MATLAB? 1.7 Discuss in brief about the GUI toolbox feature of MATLAB. 1,8 What is meant by toolbox and list some of the toolboxes available for MATLAB?Demy vpZo Introduction to © The preliminaries of Artificial Neural Networks. ‘* Definition of an artificial neuron. '* The development of neural net- works. ‘* Comparison between the biological neuron and artificial neuron based on speed, fault tolerance, memory, control mechanism, etc. © The method of setting the value for the weights and how it enables the process of learning or training. Artificial Neural © svecissins sinners Networks * Basic building blocks of the artificial neural network, i.e. network architecture, setting the weights, activation function, etc. * The activation function used to calculate the output response of a neuron. * Summary of notations used all cover in this text.Introduction to Artificial Neural Networks vt ‘The basic preliminaries involved in the Artificial Neural Network (ANN) are described in this chapter. A brief summary of the history of neural networks, in terms of the development of architectures and algorithms, the structure of the biological neuron is discussed and compared with the artificial neuron. The basic building blocks and the various terminologies of the artificial neural network are explained towards the end of the chapter. The chapter concludes by giving the summary of notations, which are used in all the network algorithms, architectures, etc. discussed in the forthcoming chapters. Artificial neural networks are nonlinear information (signal) processing devices, which are built from interconnected elementary processing devices called neurons. An Artificial Neural Network (ANN) is an information-processing paradigm that is inspired by the way biological nervous systems, such as the brain, process information. The key element of this para- digm is the novel structure of the information processing system. It is composed of a large number of highly interconnected processing elements (neurons) working in union to solve specific problems. ANNs, like people, learn by example. An ANN is configured for a specific application, such as pattern recogni- tion or data classification, through a learning process. Learning in biological systems involves adjust- ments to the synaptic connections that exist between the neurons. This is true of ANNs as well. ANN’s are a type of artificial intelligence that attempts to imitate the way a human brain works. Rather than using a digital model, in which all computations manipulate zeros and ones, a neural net- work works by creating connections between processing elements, the computer equivalent of neurons. ‘The organization and weights of the connections determine the output. ‘A neural network is a massively parallel-distributed processor that has a natural propensity for stor- ing experimental knowledge and making it available for use. It resembles the brain in two respects: 1. Knowledge is acquired by the network through a learning process, and, 2. Inter-neuron connection strengths known as synaptic weights are used to store the knowledge. Neural networks can also be defined as parameterized computational nonlinear algorithms for (nu- merical) data/signal/image processing. These algorithms are either implemented on a general-purpose ‘computer or are built into a dedicated hardware. Artificial Neural Networks thus is an information-processing system. In this information-processing system, the elements called as neurons, process the information. The signals are transmitted by means of connection links. The links possess an associated weight, which is multiplied along with the incoming signal (net input) for any typical neural net. The output signal is obtained by applying activations to the net input. An artificial neuron is characterized by: 1. Architecture (connection between neurons) 2. Training or learning (determining weights on the connections) 3. Activation function12 Introduction to Neural Networks subsections. The structure of the simple artificial neural nwt (weights) 1. Figure 2.1 shows a simple artificial neural network with two input neurons (x;, X>) and one output neuron (y). The inter connected weights are given by w, and ws. An artificial neuron is a p-input single-output signal-pro- cessing element, which can be thought of as a simple model of a non-branching biological neuron. In Fig 2.1, opt Layer various inputs to the network are represented by the math- Fig: 29 | 4 Simple Artitcial Neuf Net ematical symbol, x(n). Each of these inputs are multi- plied by a connection weight. These weights are represented by w(n). In the simplest case, these prod- ucts are simply summed, fed through a transfer function to generate a result, and then delivered as output. This process lends itself to physical implementation on a large scale in a small package. This electronic implementation is still possible with other network structures, which utilize different sum- ming functions as well as different transfer functions. Why Artificial Neural Networks? All these are discussed in detail in the forthcoming [ a network is shown in Fig. ‘The long course of evolution has given the human brain many desirable characteristics not present in Von Neumann or modern parallel computers. These include © Massive parallelism, # Distributed representation and computation, « Learning ability, * Generalization ability, * Adaptivity, © Inherent contextual information processing * Fault tolerance, and * Low energy consumption, It is hoped that devices based on biological neural networks will posses some of these desirable characteristics. Modern digital computers outperform humans in the domain of numeric computation and related symbol manipulation. However, humans can effortlessly solve complex perceptual prob- Jems (like recognizing a man in a crowd from a mere glimpse of his face) at such a high speed and extent as to dwarf the world’s fastest computer. Why is there such a remarkable difference in their perfor- mance? The biological neural system architecture is completely different from the Von Neumann archi- tecture (see Table 2.1). This difference significantly affects the type of functions each computational model can best perform. Numerous efforts to develop “intelligent” programs based on Von Neumann's centralized architec- ture have not resulted in any general-purpose intelligent programs. Inspired by biological neural net- works, ANNs are massively parallel computing systems consisting of an extremely large number of simple processors with many interconnections, ANN models attempt to use some “organizational” prin- ciples believed to be used in the human brain.Introduction to Artificial Neural Networks 13 Table 2.1] Von Neumann Computer Versus Biological Neural System complex High speed One or a few Memory Separate from a processor Localized Noncontent addressable Computing Centralized Sequential Stored programs Self-learning Raliability Very vulnerable Robust Expertise Numerical and symbolic Perceptual manipulations problems Operating Well-defined, Poorly defined, environment __well-constrained unconstrained Either humans or other computer techniques can use neural networks, with their remarkable ability to derive meaning from complicated or imprecise data, to extract patterns and detect trends that are too ‘complex to be noticed. A trained neural network can be thought of as an “expert” in the category of information it has been given to analyze. This expert can then be used to provide projections given new situations of interest and answer “what if” questions. Other advantages include: 1. Adaptive learning: An ability to learn how to do tasks based on the data given for training or initial experience, 2. Self-organization: An ANN can create its own organisation or representation of the information it receives during learning time. 3. Real-time operation: ANN computations may be carried out in parallel, using special hardware devices designed and manufactured to take advantage of this capability. 4. Fault tolerance via redundant information coding: Partial destruction of a network leads to a fing degradation of performance. However, some network capabilities may be retained even after major network damage due to this feature. The historical development of the neural networks can be traced as follows: + 1943—McCulloch and Pitts: start of the modern era of neural networks This forms a logical calculus of neural networks. A network consists of sufficient number of neu- rons (using a simple model) and properly set synaptic connections can compute any computable14 Introduction to Neural Networks function, A simple logic function is performed by a neuron in this case based upon the weights set in the McCulloch-Pitts neuron, The arrangement of neuron in this case may be represented as a combination of logic functions. The most important feature of this type of neuron is the concept of threshold. When the net input to a particular neuron is greater than the specified threshold by the user, then the neuron fires. Logic circuits are found to use this type of neurons extensively, + 1949—Hebb’s book “The organization of behavior” An explicit statement of a physiological learning rule for synaptic modification was presented for the first time. Hebb proposed that the connectivity of the brain is continually changing as an organ- ism leams differing functional tasks, and that neural assemblies are created by such changes. Hebb’s work was immensely influential among psychologists. The concept behind the Hebb theory is that if two neurons are found to be active simultaneously the strength of connection between the two neurons should be increased. This concept is similar to that of the correlation matrix learning. ‘* 1958—Rosenblatt introduces Perceptron (Block [1962], Minsky and Papert [1988]) In Perceptron network the weights on the connection paths can be adjusted. A method of iterative weight adjustment can be used in the Perceptron net. The Perceptron net is found to converge if the weights obtained allow the net to reproduce exactly all the training input and target output vector pairs + 1960—Widrow and Hoff introduce adaline ADALINE, abbreviated from Adaptive Linear Neuron uses a learning rule called as Least Mean Square rule or Delta rule. This rule is found to adjust the weights so as to reduce the difference between the net input to the output unit and the desired output. The convergence criteria in this case are the reduction of mean square error to a minimum value. This delta rule for a single layer net can be called a precursor of the backpropagation net used for multi-layer nets. The multi-layer exten- sions of Adaline formed the Madaline [Widrow and Lehr, 1990] + 1982—John Hopfield’s networks Hopfield showed how to use “Ising spin glass” type of model to store information in dynamically stable networks. His work paved the way for physicists to enter neural modeling, thereby trans- forming the field of neural networks. These nets are widely used as associative memory nets. The Hopfield nets are found to be both continuous valued and discrete valued. This net provides an efficient solution for the “Travelling Sales-man Problem’ + 1972—Kohonen’s Self-Organizing Maps (SOM) Kohonen's Self-Organizing Maps are capable of reproducing important aspects of the structure of biological neural nets. They make use of data representation using topographic maps, which are common in the nervous systems. SOM also has a wide range of applications. It shows how the output layer can pick up the correlational structure (from the inputs) in the form of the spatial arrangement of units, These nets are applied to many recognition problems. * 1985—Parker, 1986—Lecum During this period the backpropagation net paved its way into the Neural Networks. This method propagates the error information at the output units back to the hidden units using a generalized delta rule. This net is basically a multilayer, feed forward net trained by means of backpropagation. Originally, even though the work was performed by Parker (1985) the credit of publishing this net goes to Rumelhart, Hinton and Williams (1986). Backpropogation net emerged as the most popularIntroduction to Artificial Neural Networks 1s learning algorithm for the training of multilayer perceptrons and has been the workhorse for many neural network applications. + 1988—Grossberg Grossberg developed a learning rule similar to that of Kohonen, which is widely used in the Counter Propagation net. This Grossberg type of learning is also used as outstar learning. This learning occurs for all the units in a particular layer, no competition among these units is assumed. + 1987, 1990—Carpenter and Grossberg Carpenter and Grossberg invented Adaptive Resonance Theory (ART). ART was designed for both binary inputs and the continuous valued inputs. The design for the binary inputs formed ARTI, and ART2 came into being when the design became applicable to the continuous valued inputs. The most important feature of these nets is that the input patterns can be presented in any order. ‘+ 1988—Broomhead and Lowe developed Radial Basis Functions (RBF). This is also a multi- layer net that is quiet similar to the back propagation net. * 1990—Vapnik developed the support vector machine. A biological neuron or a nerve cell consists of synapses, dendrites, the cell body (or hillock), and the axon. The “building blocks” are discussed as follows: ‘* The synapses are elementary signal processing devices = A synapse is a biochemical device, which converts a pre-synaptic electrical signal into a chemical signal and then back into a post-synaptic electrical signal. «= The input pulse train has its amplitude modified by parameters stored in the synapse. The nature of this modification depends on the type of the synapse, which can be either inhibitory or excita- tory. ‘© The postsynaptic signals are aggregated and transferred along the dendrites to the nerve cell body. * The cell body generates the output neuronal signal, a spike, which is transferred along the axon to the synaptic terminals of other neurons. © The frequency of firing of a neuron is proportional to the total synaptic activities and is controlled by the synaptic parameters (weights). © The pyramidal cell can receive 104 synaptic inputs and it can fan-out the output signal to thousands of target cells—a connectivity difficult to achieve in the artificial neural networks. In general the function of the main elements can be given as, Dendrite - Receives signals from other neurons Soma Sums all the incoming signals Axon - When a particular amount of input is received, then the cell fires. It transmits signal through axon to other cells.