We consider а number of possible roles of complex dynamics and chaos in information processing by neural networks. First, we review the principles of operation for some well-known neural networks, and then discuss the approaches to using chaos in neural networks. Our main goal is to present а novel view of the problem of chaos in information processing. We demonstrate that chaos emerges naturally when a neural network forms a controlling part of a more complex system. We show that such neural networks can enhance efficiency by using chaos for explorations in a method known as Reinforcement Learning. A discussion on Hamiltonian neural networks is also presented.