Struggling to backup selfhosted fusionAuth.
-
Hi folks, new user here, and love this IAM. Its working great in all respects, however, i recently decide to test disaster recovery and spun up a local version in containers running on postgresql. I backed up everything using pg_dumpall and then tried to restore that to a new postgres instance container using psql, which it did without any errors, when I brought fusionAuth up again pointing to the restored database, it would not launch, instead it gave me, hundreds of errors in fusionauth container log like:
2025-01-31 16:33:44 fusionauth-1 | Starting fusionauth-app... 2025-01-31 16:33:44 fusionauth-1 | --> Logging to /usr/local/fusionauth/logs/fusionauth-app.log 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.600 PM INFO io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties] 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.611 PM INFO io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Set property [fusionauth-app.url] set to [http://fusionauth:9011] using configured value. 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.612 PM INFO com.inversoft.configuration.BasePropertiesFileInversoftConfiguration - 2025-01-31 16:33:44 fusionauth-1 | - Overriding default value of property [database.mysql.enforce-utf8mb4] with value [true] 2025-01-31 16:33:44 fusionauth-1 | - Overriding default value of property [FUSIONAUTH_APP_RUNTIME_MODE] with value [development] 2025-01-31 16:33:44 fusionauth-1 | - Overriding default value of property [SEARCH_TYPE] with value [elasticsearch] 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.629 PM INFO io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties] 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.629 PM INFO io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Set property [fusionauth-app.url] set to [http://fusionauth:9011] using configured value. 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.629 PM INFO com.inversoft.configuration.BasePropertiesFileInversoftConfiguration - 2025-01-31 16:33:44 fusionauth-1 | - Overriding default value of property [database.mysql.enforce-utf8mb4] with value [true] 2025-01-31 16:33:44 fusionauth-1 | - Overriding default value of property [FUSIONAUTH_APP_RUNTIME_MODE] with value [development] 2025-01-31 16:33:44 fusionauth-1 | - Overriding default value of property [SEARCH_TYPE] with value [elasticsearch] 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.682 PM INFO com.inversoft.maintenance.DefaultMaintenanceModeWorkflow - 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 16:33:44 fusionauth-1 | --------------------------------------------------------------------------------------------------------- 2025-01-31 16:33:44 fusionauth-1 | ---------------------------------- Entering Silent Configuration Mode ----------------------------------- 2025-01-31 16:33:44 fusionauth-1 | --------------------------------------------------------------------------------------------------------- 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.773 PM INFO com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [b5TD7Q] Attempting to lock database to support multi-node configurations 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.778 PM INFO com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [b5TD7Q] Obtained a database lock 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.800 PM INFO com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [b5TD7Q] Database Version [1.55.1] 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.802 PM INFO com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [b5TD7Q] Latest Migration Version [1.55.1] 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.803 PM INFO com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [b5TD7Q] Attempting to unlock database to support multi-node configurations 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.803 PM INFO com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [b5TD7Q] Unlock completed 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.804 PM INFO com.inversoft.search.ElasticRestClientHelper - Connecting to Elasticsearch at [http://search:9200] 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.996 PM INFO io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties] 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.996 PM INFO io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Set property [fusionauth-app.url] set to [http://fusionauth:9011] using configured value. 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 11:33:44.996 PM INFO com.inversoft.configuration.BasePropertiesFileInversoftConfiguration - 2025-01-31 16:33:44 fusionauth-1 | - Overriding default value of property [database.mysql.enforce-utf8mb4] with value [true] 2025-01-31 16:33:44 fusionauth-1 | - Overriding default value of property [FUSIONAUTH_APP_RUNTIME_MODE] with value [development] 2025-01-31 16:33:44 fusionauth-1 | - Overriding default value of property [SEARCH_TYPE] with value [elasticsearch] 2025-01-31 16:33:44 fusionauth-1 | 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.040 PM INFO com.inversoft.maintenance.MaintenanceModePoller - Poller started to Wait for configuration to be completed. 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.041 PM INFO io.fusionauth.app.primeframework.FusionHTTPContextAuthSetup - Initializing the FusionAuth HTTP Context. 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.071 PM INFO com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [b5TD7Q] Database Version [1.55.1] 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.071 PM INFO com.inversoft.maintenance.db.JDBCMaintenanceModeDatabaseService - [b5TD7Q] Latest Migration Version [1.55.1] 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.072 PM INFO com.inversoft.search.ElasticRestClientHelper - Connecting to Elasticsearch at [http://search:9200] 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.078 PM INFO io.fusionauth.app.guice.FusionAuthModule - 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 16:33:45 fusionauth-1 | --------------------------------------------------------------------------------------------------------- 2025-01-31 16:33:45 fusionauth-1 | --------------------------------- Starting FusionAuth version [1.55.1] ---------------------------------- 2025-01-31 16:33:45 fusionauth-1 | --------------------------------------------------------------------------------------------------------- 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.171 PM INFO io.fusionauth.api.plugin.guice.PluginModule - No plugins found 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.219 PM INFO io.fusionauth.api.service.system.NodeService - Node [ee607efc-5736-480b-b3ad-3308126a8fbb] started. 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.380 PM INFO io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Loading FusionAuth configuration file [/usr/local/fusionauth/config/fusionauth.properties] 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.380 PM INFO io.fusionauth.api.configuration.DefaultFusionAuthConfiguration - Set property [fusionauth-app.url] set to [http://fusionauth:9011] using configured value. 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.381 PM INFO com.inversoft.configuration.BasePropertiesFileInversoftConfiguration - 2025-01-31 16:33:45 fusionauth-1 | - Overriding default value of property [database.mysql.enforce-utf8mb4] with value [true] 2025-01-31 16:33:45 fusionauth-1 | - Overriding default value of property [FUSIONAUTH_APP_RUNTIME_MODE] with value [development] 2025-01-31 16:33:45 fusionauth-1 | - Overriding default value of property [SEARCH_TYPE] with value [elasticsearch] 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.381 PM INFO com.inversoft.jdbc.hikari.DataSourceProvider - Connecting to PostgreSQL database at [jdbc:postgresql://db2:5432/fusionauth] 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.384 PM INFO com.zaxxer.hikari.HikariDataSource - Database-background - Starting... 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.398 PM INFO com.zaxxer.hikari.HikariDataSource - Database-background - Start completed. 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.673 PM INFO com.inversoft.jdbc.hikari.DataSourceProvider - Connecting to PostgreSQL database at [jdbc:postgresql://db2:5432/fusionauth] 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.673 PM INFO com.zaxxer.hikari.HikariDataSource - Database-primary - Starting... 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.678 PM INFO com.zaxxer.hikari.pool.HikariPool - Database-primary - Added connection org.postgresql.jdbc.PgConnection@175f859e 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.678 PM INFO com.zaxxer.hikari.HikariDataSource - Database-primary - Start completed. 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.754 PM INFO com.inversoft.jdbc.hikari.DataSourceProvider - Connecting to PostgreSQL database at [jdbc:postgresql://db2:5432/fusionauth] 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.755 PM INFO com.zaxxer.hikari.HikariDataSource - Database-secondary - Starting... 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.759 PM INFO com.zaxxer.hikari.pool.HikariPool - Database-secondary - Added connection org.postgresql.jdbc.PgConnection@21f91efa 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.759 PM INFO com.zaxxer.hikari.HikariDataSource - Database-secondary - Start completed. 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.921 PM INFO com.inversoft.scheduler.DefaultScheduler - Starting up scheduler 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.922 PM INFO com.inversoft.scheduler.DefaultScheduler - Scheduler is running 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.952 PM ERROR com.inversoft.scheduler.LogAndRetainFailureHandler - The scheduled service [class io.fusionauth.api.service.cache.CORSConfigurationCacheLoader] failed but will be re-run. 2025-01-31 16:33:45 fusionauth-1 | org.apache.ibatis.exceptions.TooManyResultsException: Expected one result (or null) to be returned by selectOne(), but found: 2 2025-01-31 16:33:45 fusionauth-1 | at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:81) 2025-01-31 16:33:45 fusionauth-1 | at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) 2025-01-31 16:33:45 fusionauth-1 | at java.base/java.lang.reflect.Method.invoke(Method.java:580) 2025-01-31 16:33:45 fusionauth-1 | at org.apache.ibatis.session.SqlSessionManager$SqlSessionInterceptor.invoke(SqlSessionManager.java:355) 2025-01-31 16:33:45 fusionauth-1 | at jdk.proxy2/jdk.proxy2.$Proxy61.selectOne(Unknown Source) 2025-01-31 16:33:45 fusionauth-1 | at org.apache.ibatis.session.SqlSessionManager.selectOne(SqlSessionManager.java:164) 2025-01-31 16:33:45 fusionauth-1 | at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87) 2025-01-31 16:33:45 fusionauth-1 | at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:141) 2025-01-31 16:33:45 fusionauth-1 | at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86) 2025-01-31 16:33:45 fusionauth-1 | at jdk.proxy2/jdk.proxy2.$Proxy63.retrieve(Unknown Source) 2025-01-31 16:33:45 fusionauth-1 | at io.fusionauth.api.service.cache.CORSConfigurationCacheLoader.internalLoad(CORSConfigurationCacheLoader.java:36) 2025-01-31 16:33:45 fusionauth-1 | at com.inversoft.cache.BaseSingleValueCacheLoader.load(BaseSingleValueCacheLoader.java:13) 2025-01-31 16:33:45 fusionauth-1 | at io.fusionauth.api.service.cache.CORSConfigurationCacheLoader.run(CORSConfigurationCacheLoader.java:30) 2025-01-31 16:33:45 fusionauth-1 | at com.inversoft.scheduler.InjectableRunnable.run(InjectableRunnable.java:56) 2025-01-31 16:33:45 fusionauth-1 | at com.inversoft.scheduler.DefaultScheduler.<init>(DefaultScheduler.java:70) 2025-01-31 16:33:45 fusionauth-1 | at com.inversoft.scheduler.DefaultScheduler$$FastClassByGuice$$9ed33ea.GUICE$TRAMPOLINE(<generated>) 2025-01-31 16:33:45 fusionauth-1 | at com.inversoft.scheduler.DefaultScheduler$$FastClassByGuice$$9ed33ea.apply(<generated>) 2025-01-31 16:33:45 fusionauth-1 | at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82) 2025-01-31 16:33:45 fusionauth-1 | at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) 2025-01-31 16:33:45 fusionauth-1 | at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) 2025-01-31 16:33:45 fusionauth-1 | at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300) 2025-01-31 16:33:45 fusionauth-1 | at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) 2025-01-31 16:33:45 fusionauth-1 | at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169) 2025-01-31 16:33:45 fusionauth-1 | at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45) 2025-01-31 16:33:45 fusionauth-1 | at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:60) 2025-01-31 16:33:45 fusionauth-1 | at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:213) 2025-01-31 16:33:45 fusionauth-1 | at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186) 2025-01-31 16:33:45 fusionauth-1 | at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:113) 2025-01-31 16:33:45 fusionauth-1 | at com.google.inject.Guice.createInjector(Guice.java:87) 2025-01-31 16:33:45 fusionauth-1 | at com.google.inject.Guice.createInjector(Guice.java:69) 2025-01-31 16:33:45 fusionauth-1 | at com.google.inject.Guice.createInjector(Guice.java:59) 2025-01-31 16:33:45 fusionauth-1 | at org.primeframework.mvc.guice.GuiceBootstrap.initialize(GuiceBootstrap.java:60) 2025-01-31 16:33:45 fusionauth-1 | at org.primeframework.mvc.BasePrimeMain.hup(BasePrimeMain.java:69) 2025-01-31 16:33:45 fusionauth-1 | at org.primeframework.mvc.BasePrimeMain.start(BasePrimeMain.java:100) 2025-01-31 16:33:45 fusionauth-1 | at io.fusionauth.app.FusionAuthMain.main(FusionAuthMain.java:27) 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.958 PM INFO com.inversoft.search.ElasticRestClientHelper - Connecting to Elasticsearch at [http://search:9200] 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.961 PM ERROR com.inversoft.scheduler.LogAndRetainFailureHandler - The scheduled service [class io.fusionauth.api.service.cache.IdentityProviderCacheLoader] failed but will be re-run. 2025-01-31 16:33:45 fusionauth-1 | com.google.inject.ProvisionException: Unable to provision, see the following errors: 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 16:33:45 fusionauth-1 | 1) [Guice/ErrorInCustomProvider]: TooManyResultsException: Expected one result (or null) to be returned by selectOne(), but found: 2 2025-01-31 16:33:45 fusionauth-1 | at FusionAuthMVCModule.configure(FusionAuthMVCModule.java:31) 2025-01-31 16:33:45 fusionauth-1 | \_ installed by: FusionAuthModule -> Modules$OverrideModule -> FusionAuthMVCModule 2025-01-31 16:33:45 fusionauth-1 | at FusionAuthObjectMapperProvider.<init>(FusionAuthObjectMapperProvider.java:24) 2025-01-31 16:33:45 fusionauth-1 | \_ for 2nd parameter 2025-01-31 16:33:45 fusionauth-1 | while locating FusionAuthObjectMapperProvider 2025-01-31 16:33:45 fusionauth-1 | at FusionAuthContentModule.bindObjectMapper(FusionAuthContentModule.java:24) 2025-01-31 16:33:45 fusionauth-1 | \_ installed by: FusionAuthModule -> Modules$OverrideModule -> FusionAuthContentModule 2025-01-31 16:33:45 fusionauth-1 | at DefaultCipherService.<init>(DefaultCipherService.java:37) 2025-01-31 16:33:45 fusionauth-1 | \_ for 1st parameter 2025-01-31 16:33:45 fusionauth-1 | while locating DefaultCipherService 2025-01-31 16:33:45 fusionauth-1 | at DefaultReactorCore.<init>(DefaultReactorCore.java:66) 2025-01-31 16:33:45 fusionauth-1 | \_ for 1st parameter 2025-01-31 16:33:45 fusionauth-1 | while locating DefaultReactorCore 2025-01-31 16:33:45 fusionauth-1 | at InstanceManager.<init>(InstanceManager.java:38) 2025-01-31 16:33:45 fusionauth-1 | \_ for 3rd parameter 2025-01-31 16:33:45 fusionauth-1 | at DefaultReactorService.<init>(DefaultReactorService.java:90) 2025-01-31 16:33:45 fusionauth-1 | \_ for 5th parameter 2025-01-31 16:33:45 fusionauth-1 | while locating DefaultReactorService 2025-01-31 16:33:45 fusionauth-1 | at DefaultPasswordService.<init>(DefaultPasswordService.java:52) 2025-01-31 16:33:45 fusionauth-1 | \_ for 3rd parameter 2025-01-31 16:33:45 fusionauth-1 | while locating DefaultPasswordService 2025-01-31 16:33:45 fusionauth-1 | at FusionAuthConnector.<init>(FusionAuthConnector.java:43) 2025-01-31 16:33:45 fusionauth-1 | \_ for 2nd parameter 2025-01-31 16:33:45 fusionauth-1 | while locating FusionAuthConnector 2025-01-31 16:33:45 fusionauth-1 | while locating Connector annotated with @Element(setName=,uniqueId=56, type=MAPBINDER, keyType=ConnectorType) 2025-01-31 16:33:45 fusionauth-1 | at DefaultAuthenticationService.<init>(DefaultAuthenticationService.java:140) 2025-01-31 16:33:45 fusionauth-1 | \_ for 4th parameter 2025-01-31 16:33:45 fusionauth-1 | while locating DefaultAuthenticationService 2025-01-31 16:33:45 fusionauth-1 | at AppleIdentityProviderAuthenticationService.<init>(AppleIdentityProviderAuthenticationService.java:76) 2025-01-31 16:33:45 fusionauth-1 | \_ for 2nd parameter 2025-01-31 16:33:45 fusionauth-1 | while locating AppleIdentityProviderAuthenticationService 2025-01-31 16:33:45 fusionauth-1 | while locating IdentityProviderAuthenticationService annotated with @Element(setName=,uniqueId=114, type=MAPBINDER, keyType=IdentityProviderType) 2025-01-31 16:33:45 fusionauth-1 | at IdentityProviderCacheLoader.<init>(IdentityProviderCacheLoader.java:39) 2025-01-31 16:33:45 fusionauth-1 | \_ for 4th parameter 2025-01-31 16:33:45 fusionauth-1 | while locating IdentityProviderCacheLoader 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 16:33:45 fusionauth-1 | Learn more: 2025-01-31 16:33:45 fusionauth-1 | https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER 2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 16:33:45 fusionauth-1 | 1 errorThese appear to be duplicates create during restore, but i was hoping conventional backup logic would work with postgres & fusionAuth.
Im i missing something obvious? what do other folks do using pg and FA?
Thanks in advance
Franco -
@spydmobile Are all the errors specifically about the CORSConfigurationCacheLoader or are there others?
@spydmobile said in Struggling to backup selfhosted fusionAuth.:
2025-01-31 16:33:45 fusionauth-1 | 2025-01-31 11:33:45.952 PM ERROR com.inversoft.scheduler.LogAndRetainFailureHandler - The scheduled service [class io.fusionauth.api.service.cache.CORSConfigurationCacheLoader] failed but will be re-run.
2025-01-31 16:33:45 fusionauth-1 | org.apache.ibatis.exceptions.TooManyResultsException: Expected one result (or null) to be returned by selectOne(), but found: 2