Module chargés dans le broker
| Code Block |
|---|
[2019-09-17 16:49:59] INFO : [broker-dev4 ] I correctly loaded the modules: [sla,Simple-log,WebUI,Livestatus,Graphite-Perfdata] |
Création du module
| Code Block |
|---|
[2019-09-17 15:33:19] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] Creating new SLA module name : sla for broker [2019-09-17 15:33:20] INFO : [WebUI ] [sla ] [INITIALISATION ] Creating new SLA module name : sla for webui |
Parametre de connection à la base mongo
| Code Block |
|---|
[2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] Parameter load for database connection [2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - database ------------------------- :[shinken] [2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - uri ------------------------------ :[mongodb://localhost/?w=1&fsync=false] [2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - replica_set ---------------------- :[] [2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - use_ssh_tunnel ------------------- :[False] [2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - ssh_keyfile ---------------------- :[~shinken/.ssh/id_rsa] [2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - ssh_user ------------------------- :[root] [2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - use_ssh_retry_failure ------------ :[1] [2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - auto_reconnect_max_try ----------- :[3] [2019-09-17 16:49:59] INFO : [broker-dev4 ] [sla ] [INITIALISATION ] [MONGO] - auto_reconnect_sleep_between_try - :[3] |
Démarrage du process d'archive
| Code Block |
|---|
[2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] Sub process ready to work. [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] [MONGO] Parameter load for database connection [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] [MONGO] - database ------------------------- :[shinken] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] [MONGO] - uri ------------------------------ :[mongodb://localhost/?w=1&fsync=false] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] [MONGO] - replica_set ---------------------- :[] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] [MONGO] - use_ssh_tunnel ------------------- :[False] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] [MONGO] - ssh_keyfile ---------------------- :[~shinken/.ssh/id_rsa] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] [MONGO] - ssh_user ------------------------- :[root] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] [MONGO] - use_ssh_retry_failure ------------ :[1] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] [MONGO] - auto_reconnect_max_try ----------- :[3] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] [MONGO] - auto_reconnect_sleep_between_try - :[3] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] Parameter load for build sla archive [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] - time_before_shinken_inactive:[30] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] Parameter load for archiving sla [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] - size_chunk_to_archive:[10000] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] - time_between_two_chunks:[0.1] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] - keep_raw_sla_day:[7] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] - Initialisation of the archive process done [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] [MONGO] Try to open a Mongodb connection to mongodb://localhost/?w=1&fsync=false:shinken [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] [MONGO] Open mongo connection done in 0.002s [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] [MONGO] Open mongo connection done in 0.002s [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] [MONGO] Ensure mongo index done in 0.004s [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [archive (pid:13227) ] [INITIALISATION ] Load 55 elements info in cache done in 0.001s |
Archivage en cours
| Code Block |
|---|
[2019-09-18 00:08:45] INFO : [broker-dev4 ] [sla ] [archive (pid:1434) ] Archiving the day 17-09-2019 [2019-09-18 00:08:45] INFO : [broker-dev4 ] [sla ] [archive (pid:1434) ] 55 items to archive in 1 blocks ( max size block : 10000 ) [2019-09-18 00:08:45] INFO : [broker-dev4 ] [sla ] [archive (pid:1434) ] - block 0: archiving 55 elements [2019-09-18 00:08:45] INFO : [broker-dev4 ] [sla ] [archive (pid:1434) ] Found first monitoring start time at 10-09-2019 23:51:30 [2019-09-18 00:08:45] INFO : [broker-dev4 ] [sla ] [archive (pid:1434) ] - block 0: insert 55 archive in mongo done in 0.003s [2019-09-18 00:08:45] INFO : [broker-dev4 ] [sla ] [archive (pid:1434) ] - block 0: archiving block elements done in 0.043s [2019-09-18 00:08:45] INFO : [broker-dev4 ] [sla ] [archive (pid:1434) ] 55 entries archive for the day 17-09-2019 in 0.157ss nb block 1 |
Démarrage du process de migration
| Code Block |
|---|
[2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] Sub process ready to work. [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] [MONGO] Parameter load for database connection [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] [MONGO] - database ------------------------- :[shinken] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] [MONGO] - uri ------------------------------ :[mongodb://localhost/?w=1&fsync=false] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] [MONGO] - replica_set ---------------------- :[] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] [MONGO] - use_ssh_tunnel ------------------- :[False] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] [MONGO] - ssh_keyfile ---------------------- :[~shinken/.ssh/id_rsa] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] [MONGO] - ssh_user ------------------------- :[root] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] [MONGO] - use_ssh_retry_failure ------------ :[1] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] [MONGO] - auto_reconnect_max_try ----------- :[3] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] [MONGO] - auto_reconnect_sleep_between_try - :[3] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] Parameter load for migrating [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] - daily_clean_batch_size:[10000] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] - daily_clean_pause_time:[2] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] - nb_stored_days:[INFINITE] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] [MONGO] Try to open a Mongodb connection to mongodb://localhost/?w=1&fsync=false:shinken [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] [MONGO] Open mongo connection done in 0.001s [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] [MONGO] Open mongo connection done in 0.001s [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] [MONGO] Ensure mongo index done in 0.003s [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] [INITIALISATION ] Load 55 elements info in cache done in 0.001s [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [migration (pid:13226) ] check raw sla collection format collections:[Date(yday=260, year=2019):True] all is migrate:[True] |
Migration check :
Pas de migration :
| Code Block |
|---|
[2019-09-18 00:08:55] INFO : [broker-dev4 ] [sla ] [migration (pid:1433) ] No archive need migration. Check done in:0.001s. |
Démarrage d'un worker
| Code Block |
|---|
[2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] Sub process ready to work. [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] Staring new worker. [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] [MONGO] Parameter load for database connection [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] [MONGO] - database ------------------------- :[shinken] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] [MONGO] - uri ------------------------------ :[mongodb://localhost/?w=1&fsync=false] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] [MONGO] - replica_set ---------------------- :[] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] [MONGO] - use_ssh_tunnel ------------------- :[False] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] [MONGO] - ssh_keyfile ---------------------- :[~shinken/.ssh/id_rsa] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] [MONGO] - ssh_user ------------------------- :[root] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] [MONGO] - use_ssh_retry_failure ------------ :[1] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] [MONGO] - auto_reconnect_max_try ----------- :[3] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] [MONGO] - auto_reconnect_sleep_between_try - :[3] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] [MONGO] Try to open a Mongodb connection to mongodb://localhost/?w=1&fsync=false:shinken [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] [MONGO] Open mongo connection done in 0.002s [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] [MONGO] Open mongo connection done in 0.002s [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] [MONGO] Ensure mongo index done in 0.002s [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] Load previous state of 1 elements done in 0.000s. [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] Parameter load for build raw sla [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] - time_before_shinken_inactive:[30] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] - store_output:[True] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] - store_long_output:[True] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] - list_of_stored_output_status:[set([])] [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [INITIALISATION ] New worker start in 0.007s. [2019-09-17 16:50:02] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] [sla[0]]: Now running.. |
Nombre de mise à jour en base par worker par secondes
| Code Block |
|---|
[2019-09-17 16:50:11] INFO : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] 0 raw data update in database for the day 17-09-2019 |
Cas de 2 modules qui archive le même jour en base
| Code Block |
|---|
[2019-09-18 00:08:45] ERROR : [broker-dev4 ] [sla ] [archive (pid:1434) ] Fail to archive collection 150_2020 (for date 29-05-2020) because it was lock by [broker-int-13 sla2 archive (pid:23460)] and i am [broker-int-13 sla2 archive (pid:23058)] (an other process try to archive this day) |
Dans le cas où 2 modules SLA écrivent dans la même base, les 2 modules vont tenter de faire l'archive du jour en même temps. C'est pourquoi nous avons mis un lock afin d'éviter les doublons dans les entrées SLA.
Si vous avez ce log vérifiez votre configuration si vous n’avez pas 2 modules SLA ou 2 brokers qui écrivent dans la même base Mongo.
Ecriture des états futurs (les états qu'on reporte sur les jours d'après)
Lors de l'écriture des états futurs l'on a le log suivant :
| Code Block |
|---|
[2019-09-18 00:08:45] ERROR : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] Save future states in [31.00ms]. Found [10] entries in base in [0.00ms]. Saving [4] new entries in [31.00ms]. Update [0] entries in [0.00ms] |
Une fois par jour on nettoie aussi les états future à supprimer
| Code Block |
|---|
[2019-09-18 00:08:45] ERROR : [broker-dev4 ] [sla ] [Worker: 0 (pid:13266) ] The clean of future states found [4] entries to delete. There was deleted in [35.00ms (2.00ms for found entries, 33.00ms for delete entries)]. |
Log de migration des données SLA
A chaque démarrage du module on fait une recherche des données à migrer :
| Code Block |
|---|
[2020-07-22 15:11:39] INFO : [broker-master ] [sla ] [migration (pid:32233) ] Need to migrate 931144 archives |
Puis si il y a des données à migrer on log la progression :
| Code Block |
|---|
[2020-07-22 15:11:39] INFO : [broker-master ] [sla ] [migration (pid:32233) ] Migrating in progress 1000/931144 archives [2020-07-22 15:11:39] INFO : [broker-master ] [sla ] [migration (pid:32233) ] Migrating in progress 2000/931144 archives [2020-07-22 15:11:39] INFO : [broker-master ] [sla ] [migration (pid:32233) ] Migrating in progress 3000/931144 archives [2020-07-22 15:11:40] INFO : [broker-master ] [sla ] [migration (pid:32233) ] Migrating in progress 4000/931144 archives |
Puis enfin :
Si il y a eu une migration :
| Code Block |
|---|
[2020-07-22 15:18:37] INFO : [broker-master ] [sla ] [migration (pid:32233) ] Migrate 931144 archive done in:7m 0.14s. |
Sinon:
| Code Block |
|---|
[2020-07-22 09:58:47] INFO : [broker-dev4 ] [sla ] [migration (pid:10878) ] No archive need migration. Check done in:1.09ms. |
Erreurs
La connexion au serveur Mongo n'est pas établie à l'initialisation du module SLA
Avec Tunnel SSH
| Code Block | ||
|---|---|---|
| ||
[2021-02-10 10:39:45] ERROR : [ WebUI ] [SLA][INITIALISATION] Initialisation Module: Error : [ WebUI -sla ] [ SSH TUNNEL ] [ MONGODB ] - mongo connection failure : localhost:43577 ==(ssh tunnel)==> 192.168.1.87:22 ==(mongodb)==> 192.168.1.87:27017. |
Sans Tunnel SSH
| Code Block | theme | Emacs
|---|
[YYYY-MM-DD HH:MM:SS] ERROR : [ WebUI ] [SLA][INITIALISATION] Initialisation Module: Error : [ broker-master -sla ] [ MONGODB ] - mongo connection failure to 192.168.1.87:27017 |