#### For citation:

Nguyen B. H., Tsybulin V. G. Mathematical model of three competing populations and multistability of periodic regimes. *Izvestiya VUZ. Applied Nonlinear Dynamics*, 2023, vol. 31, iss. 3, pp. 316-333. DOI: 10.18500/0869-6632-003038, EDN: HHZEBK

# Mathematical model of three competing populations and multistability of periodic regimes

Purpose of this work is to analyze oscillatory regimes in a system of nonlinear differential equations describing the competition of three non-antagonistic species in a spatially homogeneous domain.

Methods. Using the theory of cosymmetry, we establish a connection between the destruction of a two-parameter family of equilibria and the emergence of a continuous family of periodic regimes. With the help of a computational experiment in MATLAB, a search for limit cycles and an analysis of multistability were carried out.

Results. We studied dynamic scenarios for a system of three competing species for different coefficients of growth and interaction. For several combinations of parameters in a computational experiment, new continuous families of limit cycles (extreme multistability) are found. We establish bistability: the coexistence of isolated limit cycles, as well as a stationary solution and an oscillatory regime.

Conclusion. We found two scenarios for locating a family of limit cycles regarding a plane passing through three equilibria corresponding to the existence of only one species. Besides cycles lying in this plane, a family is possible with cycles intersecting this plane at two points. We can consider this case as an example of periodic processes leading to overpopulation and a subsequent decline in numbers. These results will further serve as the basis for the analysis of systems of competing populations in spatially heterogeneous areas.

- Svirezhev IM, Logofet DO. Resilience of Biological Communities. Moscow: Nauka; 1978. 352 p. (in Russian).
- Murray JD. Mathematical Biology. I. An Introduction. New York: Springer; 2002. 551 p. DOI: 10.1007/b98868.
- Bazykin AD. Nonlinear Dynamics of Interacting Populations. Singapore: World Scientific; 1998. 216 p. DOI: 10.1142/2284.
- Rubin A, Riznichenko G. Mathematical Biophysics. New York: Springer; 2014. 273 p. DOI: 10. 1007/978-1-4614-8702-9.
- Frisman YY, Kulakov MP, Revutskaya OL, Zhdanova OL, Neverova GP. The key approaches and review of current researches on dynamics of structured and interacting populations. Computer Research and Modeling. 2019;11(1):119–151 (in Russian). DOI: 10.20537/2076-7633-2019-11-1- 119-151.
- Lotka AJ. Elements of Physical Biology. Philadelphia, Pennsylvania: Williams & Wilkins; 1925. 495 p.
- Volterra V. Variazioni e fluttuazioni del numero d’individui in specie animali conviventi. Memoria della Reale Accademia Nazionale dei Lincei. 1926;2:31–113 (in Italian).
- May RM, Leonard WJ. Nonlinear aspects of competition between three species. SIAM Journal on Applied Mathematics. 1975;29(2):243–253. DOI: 10.1137/0129022.
- Chia-Wei C, Lih-Ing W, Sze-Bi H. On the asymmetric May–Leonard model of three competing species. SIAM Journal on Applied Mathematics. 1998;58(1):211–226. DOI: 10.1137/S003613999 4272060.
- Antonov V, Dolicanin D, Romanovski VG, Toth J. Invariant planes and periodic oscillations in the May–Leonard asymmetric model. MATCH Communications in Mathematical and in Computer Chemistry. 2016;76(2):455–474.
- van der Hoff Q, Greeff JC, Fay TH. Defining a stability boundary for three species competition models. Ecological Modelling. 2009;220(20):2640–2645. DOI: 10.1016/j.ecolmodel.2009.07.027.
- Hou Z, Baigent S. Heteroclinic limit cycles in competitive Kolmogorov systems. Discrete & Continuous Dynamical Systems. 2013;33(9):4071–4093. DOI: 10.3934/dcds.2013.33.4071.
- Zeeman EC, Zeeman ML. An n-dimensional competitive Lotka–Volterra system is generically determined by the edges of its carrying simplex. Nonlinearity. 2002;15(6):2019–2032. DOI: 10. 1088/0951-7715/15/6/312.
- Zeeman EC, Zeeman ML. From local to global behavior in competitive Lotka-Volterra systems. Transactions of the American Mathematical Society. 2003;355(2):713–734. DOI: 10.1090/s0002- 9947-02-03103-3.
- Chen X, Jiang J, Niu L. On Lotka–Volterra equations with identical minimal intrinsic growth rate. SIAM Journal on Applied Dynamical Systems. 2015;14(3):1558–1599. DOI: 10.1137/15M1006878.
- Jiang J, Liang F. Global dynamics of 3D competitive Lotka-Volterra equations with the identical intrinsic growth rate. Journal of Differential Equations. 2020;268(6):2551–2586. DOI: 10.1016/ j.jde.2019.09.039.
- Nguyen BH, Ha DT, Tsybulin VG. Multistability for system of three competing species. Computer Research and Modeling. 2022;14(6):1325–1342 (in Russian). DOI: 10.20537/2076-7633-2022-14- 6-1325-1342.
- Yudovich VI. Bifurcations under perturbations violating cosymmetry. Doklady Physics. 2004;49(9): 522–526. DOI: 10.1134/1.1810578.
- Yudovich VI. Secondary cycle of equilibria in a system with cosymmetry, its creation by bifurcation and impossibility of symmetric treatment of it. Chaos. 1995;5(2):402–411. DOI: 10.1063/1.166110.
- Epifanov AV, Tsybulin VG. Modeling of oscillatory scenarios of the coexistence of competing populations. Biophysics. 2016;61(4):696–704. DOI: 10.1134/S0006350916040072.
- Epifanov AV, Tsybulin VG. Regarding the dynamics of cosymmetric predator – prey systems. Computer Research and Modeling. 2017;9(5):799–813 (in Russian). DOI: 10.20537/2076-7633- 2017-9-5-799-813.
- Ha TD, Tsybulin VG. Multi-stable scenarios for differential equations describing the dynamics of a predators and preys system. Computer Research and Modeling. 2020;12(6):1451–1466 (in Russian). DOI: 10.20537/2076-7633-2020-12-6-1451-1466.
- Fay TH, Greeff JC. A three species competition model as a decision support tool. Ecological Modelling. 2008;211(1–2):142–152. DOI: 10.1016/j.ecolmodel.2007.08.023.
- Bashkirtseva IA, Karpenko LV, Ryashko LB. Stochastic sensitivity of limit cycles for «predator – two preys» model. Izvestiya VUZ. Applied Nonlinear Dynamics. 2010;18(6):42–64 (in Russian). DOI: 10.18500/0869-6632-2010-18-6-42-64.
- Abramova EP, Ryazanova TV. Dynamic regimes of the stochastic “prey – predatory” model with competition and saturation. Computer Research and Modeling. 2019;11(3):515–531 (in Russian). DOI: 10.20537/2076-7633-2019-11-3-515-531.
- Bayliss A, Nepomnyashchy AA, Volpert VA. Mathematical modeling of cyclic population dynamics. Physica D: Nonlinear Phenomena. 2019;394:56–78. DOI: 10.1016/j.physd.2019.01.010.
- Frischmuth K, Budyansky AV, Tsybulin VG. Modeling of invasion on a heterogeneous habitat: taxis and multistability. Applied Mathematics and Computation. 2021;410:126456. DOI: 10.1016/ j.amc.2021.126456.
- Budyansky AV, Tsybulin VG. Modeling the dynamics of populations in a heterogeneous environment: Invasion and multistability. Biophysics. 2022;67(1):146–152. DOI: 10.1134/S0006350922010043.
- Ha TD, Tsybulin VG. Multistability for a mathematical model of the dynamics of predators and preys in a heterogeneous area. Contemporary Mathematics. Fundamental Directions. 2022;68(3): 509–521 (in Russian). DOI: 10.22363/2413-3639-2022-68-3-509-521.
- Pontryagin LS. Ordinary Differential Equations. USA: Addison-Wesley; 1962. 304 p. DOI: 10.1016/ C2013-0-01692-1.
- Waugh I, Illingworth S, Juniper M. Matrix-free continuation of limit cycles for bifurcation analysis of large thermoacoustic systems. Journal of Computational Physics. 2013;240:225–247. DOI: 10.1016/j.jcp.2012.12.034.

- 1248 reads