**Master Configuration Document -- SQL Agent Scheduler**

1.  **Summary**

This document captures the design for the DUTAS Batch Job Scheduling
system on SQL Server. It includes database architecture, schema, stored
procedures, SQL Agent job patterns, PowerShell scripts, maintenance
procedures, and examples. The aim is to provide a knowledge document for
operations, development, and handover.

2.  **Architecture Overview**

High-level components:

-   SQL Server instance: DOES-DUTAS-SQL1 hosting Devl/Test/Prod job
    scheduling databases and SQL Agent (msdb).

-   Environment databases: DevlDUTASJobSchedule, TestDUTASJobSchedule,
    ProdDUTASJobSchedule.

-   SQL Agent Jobs for each JobControl entry (3-step structure).

-   PowerShell scripts for remote job execution and logging.

-   Operator stored procedures for overrides and maintenance.

+-------------+--------------------------------------------------------+
| **          | **Description**                                        |
| Component** |                                                        |
+=============+========================================================+
| **SQL       | Centralized instance hosting all environment databases |
| Server      | and SQL Agent Jobs (DOES-DUTAS-SQL1).                  |
| Instance**  |                                                        |
+-------------+--------------------------------------------------------+
| **          | Each environment has its own job scheduling database:  |
| Databases** |                                                        |
|             | • DevlDUTASJobSchedule                                 |
|             |                                                        |
|             | • TestDUTASJobSchedule                                 |
|             |                                                        |
|             | • ProdDUTASJobSchedule                                 |
+-------------+--------------------------------------------------------+
| **SQL       | Manages the execution of scheduled jobs. Each job is   |
| Agent**     | linked to stored procedures for conditional execution  |
|             | and status tracking.                                   |
+-------------+--------------------------------------------------------+
| *           | Used for remote job triggering, job submission, and    |
| *PowerShell | maintenance automation.                                |
| Scripts**   |                                                        |
+-------------+--------------------------------------------------------+
| **Backup    | Automated PowerShell process backs up all SQL Agent    |
| Process**   | job definitions and configurations.                    |
+-------------+--------------------------------------------------------+

**Job Steps (Generic Job Template)**

Every job in SQL Agent created by usp_CreateSQLAgentJob follows 3 steps:

  ------------------------------------------------------------------------------
  **Step**         **Description**                               **Subsystem**
  ---------------- --------------------------------------------- ---------------
  **1. Check       Runs usp_CheckJobConditions_Generic \@JobName T-SQL
  Conditions**                                                   

  **2. Run Job     Executes PowerShell job file using            CmdExec
  Script**         Execute-RemoteJob.ps1 --Jobname -Env          

  **3. Update      Logs job outcome via                          T-SQL
  Execution        usp_UpdateJobStatus_Generic \@JobName         
  Status**                                                       
  ------------------------------------------------------------------------------

3.  **Database Schema**

Database: DevlDUTASJobSchedule (same schema for Test/Prod)

**3.1 Tables and DDL (key tables):**

  ------------------------------------------------------------------------------
  **Table**                 **Purpose**
  ------------------------- ----------------------------------------------------
  **JobControl**            Master list of all jobs, their type
                            (Daily/Weekly/Monthly), command behavior, active
                            status, and frequency.

  **JobDependencies**       Defines predecessor-successor relationships among
                            jobs. Used to determine sequencing and dependency
                            checks.

  **JobExitCodes**          Logs job completion codes (0 = success, 1 = failure)
                            per execution date.

  **JobExecutionHistory**   Stores audit trail of all job runs with timestamps,
                            exit codes, and trigger source.

  **FederalHolidays**       Contains all non-working dates used by scheduling
                            logic (e.g., skip runs on holidays).
  ------------------------------------------------------------------------------

**3.1.1 JobControl**

  ---------------------------------------------------------------------------
  **Column**               **Description**
  ------------------------ --------------------------------------------------
  **JobName**              Unique identifier (e.g., DEVL_DAILY_DTSRQ001).

  **ScheduledStartTime**   Scheduled start time (HH:MM format).

  **SchedulerAction**      Indicates if job should STOP or CONTINUE next in
                           chain.

  **IsActive**             1 = Enabled, 0 = Disabled.

  **Frequency**            DAILY, WEEKLY, or MONTHLY. (ONDEMAND)

  **CreatedDate**          Job Creation Date
  ---------------------------------------------------------------------------

**3.1.2 JobDependencies**

  ----------------------------------------------------------------------------
  **Column**           **Description**
  -------------------- -------------------------------------------------------
  **JobName**          The dependent job.

  **PredecessorJob**   Job that must complete successfully before current job
                       executes.
  ----------------------------------------------------------------------------

**3.1.3 JobExecutionHistory**

  ------------------------------------------------------------------------------------------
  **Column Name**          **Data Type**    **Allow   **Default /          **Description**
                                            NULLs**   Constraint**         
  ------------------------ ---------------- --------- -------------------- -----------------
  **ExecutionID**          INT (IDENTITY)   No        Primary Key          Unique identifier
                                                      (auto-increment)     for each job
                                                                           execution record.

  **JobName**              VARCHAR(50)      No        ---                  Name of the job
                                                                           as defined in
                                                                           JobControl or SQL
                                                                           Agent.

  **RunDate**              DATE             No        ---                  Scheduled date
                                                                           for the job run.

  **ScheduledStartTime**   TIME             No        ---                  Time the job was
                                                                           scheduled to
                                                                           start.

  **ActualStartTime**      DATETIME         Yes       NULL by default      The actual time
                                                                           when the job
                                                                           started.

  **ActualEndTime**        DATETIME         Yes       NULL by default      The actual time
                                                                           when the job
                                                                           completed.

  **Status**               VARCHAR(20)      Yes       Default \'Pending\', Indicates current
                                                      constrained to       execution status.
                                                      (\'Pending\',        
                                                      \'Running\',         
                                                      \'Success\',         
                                                      \'Failed\',          
                                                      \'ForceComplete\')   

  **ErrorMessage**         VARCHAR(MAX)     Yes       NULL by default      Error message
                                                                           captured for
                                                                           failed runs.

  **OverrideFlag**         BIT              Yes       Default 0            Flag indicating
                                                                           if job status was
                                                                           manually
                                                                           overridden.

  **OverrideBy**           VARCHAR(100)     Yes       NULL by default      Name of user or
                                                                           process who
                                                                           overrode the
                                                                           status.

  **OverrideDate**         DATETIME         Yes       NULL by default      Timestamp when
                                                                           the override was
                                                                           applied.
  ------------------------------------------------------------------------------------------

**3.1.4 JobExitCodes**

  -------------------------------------------------------------------------------------
  **Column Name**      **Data Type**   **Allow   **Default /        **Description**
                                       NULLs**   Constraint**       
  -------------------- --------------- --------- ------------------ -------------------
  **JobExecutionID**   INT (IDENTITY)  No        Primary Key        Unique identifier
                                                 (auto-increment)   for each job
                                                                    execution record
                                                                    logged by
                                                                    PowerShell.

  **JobName**          NVARCHAR(128)   No        ---                Name of the
                                                                    executed SQL or
                                                                    batch job.

  **RunDate**          DATE            No        ---                Date when the job
                                                                    ran.

  **ExitCode**         INT             No        ---                Exit code returned
                                                                    by PowerShell or
                                                                    job execution
                                                                    process.

  **RecordedTime**     DATETIME        No        Default GETDATE()  Timestamp when exit
                                                                    code was recorded.
  -------------------------------------------------------------------------------------

**3.1.5 FederalHoliday**

  -----------------------------------------------------------------------------------
  **Column Name**   **Data Type**  **Allow   **Default /        **Description**
                                   NULLs**   Constraint**       
  ----------------- -------------- --------- ------------------ ---------------------
  **HolidayID**     INT (IDENTITY) No        Primary Key        Unique identifier for
                                             (auto-increment)   each holiday entry.

  **HolidayDate**   DATE           No        Unique constraint  The actual date of
                                                                the federal holiday.

  **HolidayName**   VARCHAR(100)   No        ---                The name of the
                                                                holiday (e.g.,
                                                                "Independence Day").

  **Year**          INT            No        ---                Calendar year the
                                                                holiday applies to.
  -----------------------------------------------------------------------------------

**3.2 Stored Procedures (core) - Source Code & Purpose**

Below are the finalized stored procedures that implement the logic for
the scheduling system. This section includes the purpose and the final
tested T-SQL for each one.

  ------------------------------------------------------------------------------
  **Stored Procedure**                 **Description**
  ------------------------------------ -----------------------------------------
  **usp_CreateSQLAgentJob**            Creates a new SQL Agent job dynamically
                                       from JobControl entries. Default run on
                                       Monday to Friday (modify schedule if
                                       needed manually).

  **usp_ScheduleMonthlyJob**           Dynamically attaches or reattaches
                                       one-time monthly job schedules using
                                       \@Env, \@RunDate, and \@RunTime.

  **usp_CheckJobConditions_Generic**   Evaluates predecessor job statuses and
                                       environmental conditions before running a
                                       job.

  **usp_UpdateJobStatus_Generic**      Updates the job execution outcome
                                       (Success / Failure) in JobControl after
                                       each run.

  **usp_TriggerMissedJobs**            Checks for missed executions and triggers
                                       them if required.

  **usp_ForceComplete**                Utility to close out jobs manually when
                                       needed for maintenance or re-runs.
  ------------------------------------------------------------------------------

**3.3 PowerShell Scripts (key)**

**3.3.1 Execute-RemoteJob.ps1**

Purpose: Wrapper that invokes remote job and logs exit code to
JobExitCodes table.

**3.3.2 RCSubmit-Job.ps1**

Purpose: Runs legacy submit command and evaluates SYSLOG and exit codes.
Customize acceptance list per job.

**3.3.3 Backup-SQLAgentJobs.ps1**

Purpose: Exports all SQL Agent jobs as .sql files and zips them. (Used
for job backups prior to promotion)

**4. Operational Procedures**

**4.1 Standard Daily Checklist (before 4:00 AM):**

-   Verify SQL Server and SQL Agent services are running.

-   Verify JobControl entries and schedules.

**4.2 Ad-hoc Maintenance / Override Flow:**

1.  If a job fails (STOP), investigate and fix the root cause.

2.  Re-run the failed job PowerShell or via SSMS.

3.  Run SQL: ForceCompleteJob.sql to mark job force complete. It uses
    Stored Procedure: usp_ForceComplete and takes Job Name and Operator
    Name as input.

4.  After ad-hoc operations, you can run the jobs that missed scheduled
    time start either one by one or you can run
    dbo.usp_TriggerMissedJobs to trigger jobs that missed their
    scheduled start while failed job resolution was in progress.

Sample commands:

\-- Force Complete (example)\
EXEC dbo.usp_ForceComplete \'DEVL_WEEKLY_DTSGSID1\', \'Neeraj.Kumar\';

\-- 1. Preview missed jobs (safe, no run):

EXEC dbo.usp_TriggerMissedJobs \@DryRun = 1;

\-- 2. Actually trigger missed jobs:

EXEC dbo.usp_TriggerMissedJobs \@DryRun = 0;

**4.3 Monthly Job Scheduling and Execution Process**

Monthly jobs in the the SQL Agent are designed to run on-demand or on
specific monthly dates rather than on a recurring automated date.

To maintain control and prevent unintentional executions, all monthly
jobs are kept disabled by default (IsActive = 0) and are only activated
when needed.

  ------------------------------------------------------------------------------
  **Step**   **Action**    **Script / Procedure Used**        **Purpose**
  ---------- ------------- ---------------------------------- ------------------
  1          Review        SELECT \* FROM JobControl WHERE    Identify jobs to
             monthly job   Frequency=\'MONTHLY\';             be run this month.
             list                                             

  2          Update run    Edit ActivateMonthlyJob.sql        Define monthly
             date & time                                      execution window.

  3          Schedule &    Run ActivateMonthlyJob.sql         Create one-time
             activate jobs                                    SQL Agent
                                                              schedules.

  4          Monitor       Use JobExecutionHistory table or   Track progress and
             execution     SQL Agent history                  outcomes.

  5          Deactivate    Run DeactivateMonthlyJob.sql       Reset jobs to
             jobs post-run                                    inactive state.
  ------------------------------------------------------------------------------

**4.3.1 Default State: Jobs Disabled**

-   All monthly jobs are initialized in the JobControl table with
    IsActive = 0.

-   This ensures that they do not run automatically unless explicitly
    scheduled.

-   Frequency is stored as \'MONTHLY\' for identification, but
    scheduling depends entirely on the stored procedure.

Example Query:

SELECT JobName, IsActive, Frequency

FROM dbo.JobControl

WHERE Frequency = \'MONTHLY\';

**4.3.2 Activating and Scheduling Monthly Jobs**

To schedule monthly jobs for the upcoming month (e.g., October or
November), you'll use the ActivateMonthlyJob.sql script.

-   Updates job activation flags (IsActive = 1).

-   Calls the stored procedure usp_ScheduleMonthlyJob to dynamically
    create or refresh SQL Agent schedules.

-   Removes any old schedules and replaces them with new, one-time
    schedules for the provided date and time.

> Example Query:
>
> EXEC dbo.usp_ScheduleMonthlyJob
>
> \@JobName = \'DEVL_MONTHLY_DTSCHGVB\',
>
> \@Env = \'DEVL\',
>
> \@RunDate = \'2025-10-31\',
>
> \@RunTime = \'19:26:00\';

What it does internally:

-   Identifies the correct environment database:

    -   DEVL → DevlDUTASJobSchedule

    -   TEST → TestDUTASJobSchedule

    -   PROD → ProdDUTASJobSchedule

-   Sets the job as active in JobControl.

-   Detaches and deletes any existing schedule for the job.

-   Creates a **new one-time SQL Agent schedule** for the given
    date/time.

-   Reattaches the schedule to the job.

-   Prints confirmation once the job is scheduled.

> **4.3.3 Post-Run Deactivation**
>
> After monthly jobs finish executing successfully:

-   Deactivate them again to prevent accidental re-runs.

-   This is done using the DeactivateMonthlyJob.sql script.

Example Query:

UPDATE dbo.JobControl

SET IsActive = 0

WHERE Frequency = \'MONTHLY\';
