Wrong PersistentStore is used when creating multiple Schedulers

[Roiskia]

Describe the bug

I am trying to instantiate multiple, independent schedulers with their own persistent stores. But after creating more then one, they all seem to use the same.

I am not sure what I am doing something wrong or if I have missed something in the documentation about this and would appreciate any advice.

Version used

Quartz.Net 3.7.0

To Reproduce

I parse a list of configured datapools, make sure they are unique, and then instantiate a scheduler for each of them like this:

string datapool = "example";

var factory = SchedulerBuilder.Create()
    .WithId(Guid.NewGuid().ToString())
    .WithName(datapool)
    .UsePersistentStore(opt => GetConStr(opt, datapool))
    .Build();

var scheduler = await factory.GetScheduler();

The GetConStr method determines what type of database the datapool is using and calls the appropriate extension method on the PersistantStoreOptions object with the datapool's connection string (such as UseOracle/UseSqlServer).

For example, Datapool A might use SQL Server and Datapool B might use Oracle.
Using only Datapool A will write to the SQL Server connection.
Using only Datapool B will write to the Oracle connection.
Using both A and B in this order will result in both writing to the Oracle connection.

Expected behavior

Using both Datapools A and B in this order will cause the first scheduler to write to SQL Server und the second one to write to Oracle.

[lahma]

Can you create a complete sample, this is just one configuration. You can create check what kind of configuration you have with:

var pool1 = SchedulerBuilder.Create()
    .WithId(Guid.NewGuid().ToString())
    .WithName("pool1");

foreach (string key in pool1.Properties.Keys)
{
    Console.WriteLine($"{key}: {pool1.Properties[key]}");
}

SchedulerBuilder is just a strongly-typed wrapper over generic key-value configuration.

[Roiskia]

Sure, here is a more complete example of what I am doing. As explained earlier, I am creating a scheduler for each unique datapool.
The log output seems to be correct at this point.

foreach (var datapool in datapools)
{
    if (_datapools.ContainsKey(datapool))
    {
        continue;
    }

    var builder = SchedulerBuilder.Create()
    .WithId(Guid.NewGuid().ToString())
    .WithName(datapool)
    .UsePersistentStore(opt => GetConStr(opt, datapool))
    
    StringBuilder stringBuilder = new StringBuilder();
    stringBuilder.AppendLine("Scheduler config:");
    foreach(string key in builder.Properties.Keys)
    {
        stringBuilder.AppendLine($"{key}: {builder.Properties[key]}");
    }
    var config = stringBuilder.ToString();
    _logger.Debug(config);
    
    var factory = builder.Build();

    var scheduler = await factory.GetScheduler();
    
    _schedulers.Add(datapool, scheduler);
}

[lahma]

So what's the output? Please hide credentials and other sensitive data from connection strings.

[Roiskia]

The output looks like this:

quartz.scheduler.instanceId: a9631b10-7706-4879-bf1d-e68de08d233a
quartz.scheduler.instanceName: datapool-a
quartz.jobStore.type: Quartz.Impl.AdoJobStore.JobStoreTX, Quartz
quartz.jobStore.useProperties: true
quartz.jobStore.driverDelegateType: Quartz.Impl.AdoJobStore.SqlServerDelegate, Quartz
quartz.jobStore.dataSource: default
quartz.dataSource.default.provider: SqlServer
quartz.dataSource.default.connectionString: data source=********;password=********;user id=********;
quartz.serializer.type: Quartz.Simpl.JsonObjectSerializer, Quartz.Serialization.Json
quartz.jobStore.misfireThreshold: 60000
quartz.scheduler.interruptJobsOnShutdown: True
quartz.scheduler.interruptJobsOnShutdownWithWait: True
quartz.scheduler.batchTriggerAcquisitionFireAheadTimeWindow: 1

quartz.scheduler.instanceId: fe07fdaa-2b71-4816-9cf2-049f60d40d9a
quartz.scheduler.instanceName: datapool-b
quartz.jobStore.type: Quartz.Impl.AdoJobStore.JobStoreTX, Quartz
quartz.jobStore.useProperties: true
quartz.jobStore.driverDelegateType: Quartz.Impl.AdoJobStore.OracleDelegate, Quartz
quartz.jobStore.dataSource: default
quartz.dataSource.default.provider: OracleODPManaged
quartz.dataSource.default.connectionString: data source=********;password=********;user id=********;
quartz.serializer.type: Quartz.Simpl.JsonObjectSerializer, Quartz.Serialization.Json
quartz.jobStore.misfireThreshold: 60000
quartz.scheduler.interruptJobsOnShutdown: True
quartz.scheduler.interruptJobsOnShutdownWithWait: True
quartz.scheduler.batchTriggerAcquisitionFireAheadTimeWindow: 1

[lahma]

Seems that configuration have different values so calling builder.Build() for each should create different scheduler factories producing schedulers with different dbs.

[Roiskia]

I understand that. And I expected that. But I can reliably reproduce the problem described.

If I build schedulers A and B in this order, A will write with the connection from B.
I also get data with both SCHED_NAME values in B.

If I build B first and then A, I end up with the same problem, but now both are writing to A.

Even though I get the correct config output for each scheduler.

If I build only A, it will write to A. If I build only B, it will write to B.

[Roiskia]

I have prepared a sample project (see attached zip) that reproduces the problem for you. For simplicity, I am trying to connect to two different Oracle instances in this case.

QuartzExample.zip

[Roiskia]

Today I noticed a github notification for a post by Uwe Laas on this topic, written on 13.12.2023, which is no longer there. Or at least I cannot see it. Is this the root of the problem or does it no longer apply?

Sorry to jump in here, had the same problem while writing tests - isn't it the case that the SchedulerFactory uses the static 'Instance' of SchedulerRepository? I thought it was impossible to create multiple different schedulers within the same process, at least without using dirty tricks like reflection. So, always the same scheduler => always the same store. Or am I wrong?

[JezhikLaas]

Sorry, I deleted my comment because I completely misunderstood the problem. Am 15.12.23 um 09:30 schrieb Roiskia: Today I noticed a github notification for a post by Uwe Laas on this topic, written on 13.12.2023, which is no longer there. Or at least I cannot see it. Is this the root of the problem or does it no longer apply? Sorry to jump in here, had the same problem while writing tests - isn't it the case that the SchedulerFactory uses the static 'Instance' of SchedulerRepository? I thought it was impossible to create multiple different schedulers within the same process, at least without using dirty tricks like reflection. So, always the same scheduler => always the same store. Or am I wrong? — Reply to this email directly, view it on GitHub<#2218 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AG5DAI2HZGO45J3K6EMRJG3YJQDC3AVCNFSM6AAAAABARRAGKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJXGQ3TQMZRGM>. You are receiving this because you commented.Message ID: ***@***.***>

…

-- Uwe Laas Tel 05221 1718677 Laas IT Consulting https://www.laas-it-consulting.de

[Roiskia]

This is fine. I just wanted to ask because it sounded like a plausible cause of the problem.

Sorry, I deleted my comment because I completely misunderstood the problem.

[Roiskia]

Today i managed to debug the whole process and it seems that the Quartz.Util.DBConnectionManager singleton is the problem here. As the factories within the process will all be using the same internal provider dictionary, they will also be overriding each other's default provider. If I set a dataSourceName on the UseX methods, I can avoid this for now.

The only problem I have here is that this is not obvious at all. Since I am creating new factories, I would not expect them to share any state.