Overview

 The TaskQueueScheduler module delivers flexible, dynamic scheduling of microflows on available taskqueues while avoiding hardcoded (in Mendix Studio) ScheduledEvents and TaskQueue assignments. Via same UI it also allows running and tracking microflows in batch (on Batch taskqueue)

Documentation

Depends on Mendix 9 or higher as that introduced TaskQueues

As of version 1.2.7 (RunAsUser feature) it also depends on CommunityCommons and System.User

As of version 1.3.0 it also depends on OQL module (developed with OQL v2.9, tested with OQL v2.12) 

OQL v2.2 gave null pointer exception, which was resolved by upgrading to OQL v2.12

IMPORTANT: Install those from MarketPlace first when they are not yet in your current project

Removed dependency on MxModelReflection module now using metamodel to get microflows without input parameter (to run on schedule queues)

Removed dependency on Validate module while all validations are still found in Validations folder

This TaskQueueScheduler model is developed to replace old MicroflowScheduler and ProcessQueue modules from AppStore

in a Mendix 9 compliant way, because both became deprecated with Mendix 9.

Microflow scheduling functionality:

Module delivers flexible, dynamic scheduling of microflows so we can avoid most hardcoded (in Mendix Studio) ScheduledEvents.

The Schedule handler (scheduled every minute) now puts microflows on the specified Schedule TaskQueue when specified runtime

is reached.

ProcessQueue functionality:

When a microflow is started on a Taskqueue via Schedule the Schedule_Overview page shows Queued/Running, Finished and

Cancelled tabs to track running microflows. Also it shows a Configuration tab where you can dynamically set/update the

TaskQueue a microflow is to run on. It is not required to preconfigure a microflow on a TaskQueue upfront (like in ProcessQueue).

A new (yet unknown) microflow started via Schedule will be added and run on default Batch TaskQueue automatically.

Via Configuration tab you can also directly run (non-scheduled) microflows without input parameters.

On QueuedAction:

As we run microflows on TaskQueue directly there is no longer a need to create a QueuedAction to do that.

QueuedAction are only used to run a microflow on batch chunks.

In Running tab you see how many batch chuncks are still on the queue.

The microflow stays running till all its batch chunks (QueueActions) are done.

On Finished tab you'll find the timing for each batch chunk of the microflow.

Please check _SAMPLES folder on how to do this.

• All Schedules: batch (QueuedAction) per object
• All Schedules: batches (QueuedAction) based on BatchSize and Count
• All Schedules: batches (single Microflow, no QueuedAction) based on BatchSize and Count
• All Schedules (fast): batches based on GUID and BatchSize
• All Schedules (fast): batches based on GUID incremental
• Update Schedules: Sequential update batches of BatchSize (Batch1Clusterwide queue)

Schedule_Overview page Configuration tab has a Samples groupbox from which you can run the samples.

QueuedActions are deleted on completion as we have no need for them after that.

To conclude the TaskQueueScheduler module now delivers flexible, dynamic scheduling of microflows on available taskqueues

while avoiding hardcoded (in Mendix Studio) ScheduledEvents and TaskQueue assignments.

How to configure:

Download TaskQueueScheduler module from Appstore into your application.

Add Schedule_Overview page to your app's navigation.

Pages have Atlas_Default layout you probably like to change that to your app's layout.

Open Schedule_Overview-Confuguration and hit to initialize TaskQueue objects.

You'll also need to go there when you add/change Mendix TaskQueues in your application

YML adjustments so not all ScheduledEvents from Studio keep on running after switching to taskqueue scheduler:

# * * * Scheduled Events * * *

# ALL, NONE or comma delimited .

# no spaces in between SCHEDULED_EVENTS:

TaskQueueScheduler.ScE_SchedulerQueues

Updating to newer version:

In most cases upgrade should work smoothly but be extra careful when REPLACING existing module as then you could LOOSE your carefully scheduled microflows.

Suggested update procedure:

1. Export your current Schedule to an Excel file

2. Now replace TaskQueueScheduler module with newer version

3. Import Excel file to restore all Schedules

Updating to v1.2.7: all should still work, just make sure you configure Batch1Clusterwide queue in TaskQueue_overview page

Updating to v1.3.0: make sure you have installed OQL module (from MarketPlace) before installing 

How to use:

By default there are 3 Schedule and 3 Batch TaskQueues defined, but you can add more when desired. 

Schedule queues are for scheduling microflows

Batch is the default queue for batch processing

Batch3Threads is for high speed parallel batch processing, for simple fast processes only.

Batch1Clusterwide is for sequential 1 by 1 processing. Not fast but it garantees microflows are executed one after the other even in multi instance environments.

Additional TaskQueue(s) can be created in any module, just remember to also update theTaskQueue objects to the new situation

via Schedule-Configuration button.

For each TaskQueue object you can set AllowScheduling or not (default)

Via Schedule-Configuration you can run microflows without input parameters on the fly.

Via microflow you can start a schedule with QueuedAction object input which allows for running big updates in small batches

or just specifying an other input object.

Starting microflows on a queue via StudioPro will work as always, but then the microflow will not appear in Queued/Running tab

and you do not have the option to change the TaskQueue dynamically

Schedule ACT_CleanProcessedQueueTask regularly to prevent unwanted growth of System.ProcessedQueueTask

Concurrency on a single thread taskqueue:

With multiple instances indeed microflows are picked up by every instance so there can be parallel execution (on different instance)

even with single thread taskqueues.

To avoid parallel execution with multiple instances you should start your microflow via ACT_RunMicroflowOnQueue_ByName_Once

as then Schedule will prevent same microflow being put on taskqueue again

Alternative: Define a taskqueue as ClusterWide 1 thread.

Mendix arranges tasks put on such queue to be execute after each other (even in multicluster environments)

Contineous deploy issue:

ASU_ microflow calls another microflow on a taskqueue to avoid longrunning ASU processing.

During deploy the CANDIDATE CF app runs the ASU microflow, but taskqueues run on all instances

so Mendix can process the queued microflow on an old (but still running) instance

of the app.

Problem: the old instance can have an outdated version of the queued microflow still.

Solution: Call ACT_PauseSchedule as first action in the main ASU_ microflow

Like this the queued microflows are only initiated when old instances are stopped.

Creating scheduled microflows automatically:

Call SUB_ScheduleMicroflow to create a scheduled microflow.

When microflow is already scheduled it will do nothing.

This particularly convenient to automatically check if all necessary microflows in your project are scheduled