431 lines
19 KiB
Markdown
431 lines
19 KiB
Markdown
**Master Configuration Document -- SQL Agent Scheduler**
|
|
|
|
1. **Summary**
|
|
|
|
This document captures the design for the DUTAS Batch Job Scheduling
|
|
system on SQL Server. It includes database architecture, schema, stored
|
|
procedures, SQL Agent job patterns, PowerShell scripts, maintenance
|
|
procedures, and examples. The aim is to provide a knowledge document for
|
|
operations, development, and handover.
|
|
|
|
2. **Architecture Overview**
|
|
|
|
High-level components:
|
|
|
|
- SQL Server instance: DOES-DUTAS-SQL1 hosting Devl/Test/Prod job
|
|
scheduling databases and SQL Agent (msdb).
|
|
|
|
- Environment databases: DevlDUTASJobSchedule, TestDUTASJobSchedule,
|
|
ProdDUTASJobSchedule.
|
|
|
|
- SQL Agent Jobs for each JobControl entry (3-step structure).
|
|
|
|
- PowerShell scripts for remote job execution and logging.
|
|
|
|
- Operator stored procedures for overrides and maintenance.
|
|
|
|
+-------------+--------------------------------------------------------+
|
|
| ** | **Description** |
|
|
| Component** | |
|
|
+=============+========================================================+
|
|
| **SQL | Centralized instance hosting all environment databases |
|
|
| Server | and SQL Agent Jobs (DOES-DUTAS-SQL1). |
|
|
| Instance** | |
|
|
+-------------+--------------------------------------------------------+
|
|
| ** | Each environment has its own job scheduling database: |
|
|
| Databases** | |
|
|
| | • DevlDUTASJobSchedule |
|
|
| | |
|
|
| | • TestDUTASJobSchedule |
|
|
| | |
|
|
| | • ProdDUTASJobSchedule |
|
|
+-------------+--------------------------------------------------------+
|
|
| **SQL | Manages the execution of scheduled jobs. Each job is |
|
|
| Agent** | linked to stored procedures for conditional execution |
|
|
| | and status tracking. |
|
|
+-------------+--------------------------------------------------------+
|
|
| * | Used for remote job triggering, job submission, and |
|
|
| *PowerShell | maintenance automation. |
|
|
| Scripts** | |
|
|
+-------------+--------------------------------------------------------+
|
|
| **Backup | Automated PowerShell process backs up all SQL Agent |
|
|
| Process** | job definitions and configurations. |
|
|
+-------------+--------------------------------------------------------+
|
|
|
|
**Job Steps (Generic Job Template)**
|
|
|
|
Every job in SQL Agent created by usp_CreateSQLAgentJob follows 3 steps:
|
|
|
|
------------------------------------------------------------------------------
|
|
**Step** **Description** **Subsystem**
|
|
---------------- --------------------------------------------- ---------------
|
|
**1. Check Runs usp_CheckJobConditions_Generic \@JobName T-SQL
|
|
Conditions**
|
|
|
|
**2. Run Job Executes PowerShell job file using CmdExec
|
|
Script** Execute-RemoteJob.ps1 --Jobname -Env
|
|
|
|
**3. Update Logs job outcome via T-SQL
|
|
Execution usp_UpdateJobStatus_Generic \@JobName
|
|
Status**
|
|
------------------------------------------------------------------------------
|
|
|
|
3. **Database Schema**
|
|
|
|
Database: DevlDUTASJobSchedule (same schema for Test/Prod)
|
|
|
|
**3.1 Tables and DDL (key tables):**
|
|
|
|
------------------------------------------------------------------------------
|
|
**Table** **Purpose**
|
|
------------------------- ----------------------------------------------------
|
|
**JobControl** Master list of all jobs, their type
|
|
(Daily/Weekly/Monthly), command behavior, active
|
|
status, and frequency.
|
|
|
|
**JobDependencies** Defines predecessor-successor relationships among
|
|
jobs. Used to determine sequencing and dependency
|
|
checks.
|
|
|
|
**JobExitCodes** Logs job completion codes (0 = success, 1 = failure)
|
|
per execution date.
|
|
|
|
**JobExecutionHistory** Stores audit trail of all job runs with timestamps,
|
|
exit codes, and trigger source.
|
|
|
|
**FederalHolidays** Contains all non-working dates used by scheduling
|
|
logic (e.g., skip runs on holidays).
|
|
------------------------------------------------------------------------------
|
|
|
|
**3.1.1 JobControl**
|
|
|
|
---------------------------------------------------------------------------
|
|
**Column** **Description**
|
|
------------------------ --------------------------------------------------
|
|
**JobName** Unique identifier (e.g., DEVL_DAILY_DTSRQ001).
|
|
|
|
**ScheduledStartTime** Scheduled start time (HH:MM format).
|
|
|
|
**SchedulerAction** Indicates if job should STOP or CONTINUE next in
|
|
chain.
|
|
|
|
**IsActive** 1 = Enabled, 0 = Disabled.
|
|
|
|
**Frequency** DAILY, WEEKLY, or MONTHLY. (ONDEMAND)
|
|
|
|
**CreatedDate** Job Creation Date
|
|
---------------------------------------------------------------------------
|
|
|
|
**3.1.2 JobDependencies**
|
|
|
|
----------------------------------------------------------------------------
|
|
**Column** **Description**
|
|
-------------------- -------------------------------------------------------
|
|
**JobName** The dependent job.
|
|
|
|
**PredecessorJob** Job that must complete successfully before current job
|
|
executes.
|
|
----------------------------------------------------------------------------
|
|
|
|
**3.1.3 JobExecutionHistory**
|
|
|
|
------------------------------------------------------------------------------------------
|
|
**Column Name** **Data Type** **Allow **Default / **Description**
|
|
NULLs** Constraint**
|
|
------------------------ ---------------- --------- -------------------- -----------------
|
|
**ExecutionID** INT (IDENTITY) No Primary Key Unique identifier
|
|
(auto-increment) for each job
|
|
execution record.
|
|
|
|
**JobName** VARCHAR(50) No --- Name of the job
|
|
as defined in
|
|
JobControl or SQL
|
|
Agent.
|
|
|
|
**RunDate** DATE No --- Scheduled date
|
|
for the job run.
|
|
|
|
**ScheduledStartTime** TIME No --- Time the job was
|
|
scheduled to
|
|
start.
|
|
|
|
**ActualStartTime** DATETIME Yes NULL by default The actual time
|
|
when the job
|
|
started.
|
|
|
|
**ActualEndTime** DATETIME Yes NULL by default The actual time
|
|
when the job
|
|
completed.
|
|
|
|
**Status** VARCHAR(20) Yes Default \'Pending\', Indicates current
|
|
constrained to execution status.
|
|
(\'Pending\',
|
|
\'Running\',
|
|
\'Success\',
|
|
\'Failed\',
|
|
\'ForceComplete\')
|
|
|
|
**ErrorMessage** VARCHAR(MAX) Yes NULL by default Error message
|
|
captured for
|
|
failed runs.
|
|
|
|
**OverrideFlag** BIT Yes Default 0 Flag indicating
|
|
if job status was
|
|
manually
|
|
overridden.
|
|
|
|
**OverrideBy** VARCHAR(100) Yes NULL by default Name of user or
|
|
process who
|
|
overrode the
|
|
status.
|
|
|
|
**OverrideDate** DATETIME Yes NULL by default Timestamp when
|
|
the override was
|
|
applied.
|
|
------------------------------------------------------------------------------------------
|
|
|
|
**3.1.4 JobExitCodes**
|
|
|
|
-------------------------------------------------------------------------------------
|
|
**Column Name** **Data Type** **Allow **Default / **Description**
|
|
NULLs** Constraint**
|
|
-------------------- --------------- --------- ------------------ -------------------
|
|
**JobExecutionID** INT (IDENTITY) No Primary Key Unique identifier
|
|
(auto-increment) for each job
|
|
execution record
|
|
logged by
|
|
PowerShell.
|
|
|
|
**JobName** NVARCHAR(128) No --- Name of the
|
|
executed SQL or
|
|
batch job.
|
|
|
|
**RunDate** DATE No --- Date when the job
|
|
ran.
|
|
|
|
**ExitCode** INT No --- Exit code returned
|
|
by PowerShell or
|
|
job execution
|
|
process.
|
|
|
|
**RecordedTime** DATETIME No Default GETDATE() Timestamp when exit
|
|
code was recorded.
|
|
-------------------------------------------------------------------------------------
|
|
|
|
**3.1.5 FederalHoliday**
|
|
|
|
-----------------------------------------------------------------------------------
|
|
**Column Name** **Data Type** **Allow **Default / **Description**
|
|
NULLs** Constraint**
|
|
----------------- -------------- --------- ------------------ ---------------------
|
|
**HolidayID** INT (IDENTITY) No Primary Key Unique identifier for
|
|
(auto-increment) each holiday entry.
|
|
|
|
**HolidayDate** DATE No Unique constraint The actual date of
|
|
the federal holiday.
|
|
|
|
**HolidayName** VARCHAR(100) No --- The name of the
|
|
holiday (e.g.,
|
|
"Independence Day").
|
|
|
|
**Year** INT No --- Calendar year the
|
|
holiday applies to.
|
|
-----------------------------------------------------------------------------------
|
|
|
|
**3.2 Stored Procedures (core) - Source Code & Purpose**
|
|
|
|
Below are the finalized stored procedures that implement the logic for
|
|
the scheduling system. This section includes the purpose and the final
|
|
tested T-SQL for each one.
|
|
|
|
------------------------------------------------------------------------------
|
|
**Stored Procedure** **Description**
|
|
------------------------------------ -----------------------------------------
|
|
**usp_CreateSQLAgentJob** Creates a new SQL Agent job dynamically
|
|
from JobControl entries. Default run on
|
|
Monday to Friday (modify schedule if
|
|
needed manually).
|
|
|
|
**usp_ScheduleMonthlyJob** Dynamically attaches or reattaches
|
|
one-time monthly job schedules using
|
|
\@Env, \@RunDate, and \@RunTime.
|
|
|
|
**usp_CheckJobConditions_Generic** Evaluates predecessor job statuses and
|
|
environmental conditions before running a
|
|
job.
|
|
|
|
**usp_UpdateJobStatus_Generic** Updates the job execution outcome
|
|
(Success / Failure) in JobControl after
|
|
each run.
|
|
|
|
**usp_TriggerMissedJobs** Checks for missed executions and triggers
|
|
them if required.
|
|
|
|
**usp_ForceComplete** Utility to close out jobs manually when
|
|
needed for maintenance or re-runs.
|
|
------------------------------------------------------------------------------
|
|
|
|
**3.3 PowerShell Scripts (key)**
|
|
|
|
**3.3.1 Execute-RemoteJob.ps1**
|
|
|
|
Purpose: Wrapper that invokes remote job and logs exit code to
|
|
JobExitCodes table.
|
|
|
|
**3.3.2 RCSubmit-Job.ps1**
|
|
|
|
Purpose: Runs legacy submit command and evaluates SYSLOG and exit codes.
|
|
Customize acceptance list per job.
|
|
|
|
**3.3.3 Backup-SQLAgentJobs.ps1**
|
|
|
|
Purpose: Exports all SQL Agent jobs as .sql files and zips them. (Used
|
|
for job backups prior to promotion)
|
|
|
|
**4. Operational Procedures**
|
|
|
|
**4.1 Standard Daily Checklist (before 4:00 AM):**
|
|
|
|
- Verify SQL Server and SQL Agent services are running.
|
|
|
|
- Verify JobControl entries and schedules.
|
|
|
|
**4.2 Ad-hoc Maintenance / Override Flow:**
|
|
|
|
1. If a job fails (STOP), investigate and fix the root cause.
|
|
|
|
2. Re-run the failed job PowerShell or via SSMS.
|
|
|
|
3. Run SQL: ForceCompleteJob.sql to mark job force complete. It uses
|
|
Stored Procedure: usp_ForceComplete and takes Job Name and Operator
|
|
Name as input.
|
|
|
|
4. After ad-hoc operations, you can run the jobs that missed scheduled
|
|
time start either one by one or you can run
|
|
dbo.usp_TriggerMissedJobs to trigger jobs that missed their
|
|
scheduled start while failed job resolution was in progress.
|
|
|
|
Sample commands:
|
|
|
|
\-- Force Complete (example)\
|
|
EXEC dbo.usp_ForceComplete \'DEVL_WEEKLY_DTSGSID1\', \'Neeraj.Kumar\';
|
|
|
|
\-- 1. Preview missed jobs (safe, no run):
|
|
|
|
EXEC dbo.usp_TriggerMissedJobs \@DryRun = 1;
|
|
|
|
\-- 2. Actually trigger missed jobs:
|
|
|
|
EXEC dbo.usp_TriggerMissedJobs \@DryRun = 0;
|
|
|
|
**4.3 Monthly Job Scheduling and Execution Process**
|
|
|
|
Monthly jobs in the the SQL Agent are designed to run on-demand or on
|
|
specific monthly dates rather than on a recurring automated date.
|
|
|
|
To maintain control and prevent unintentional executions, all monthly
|
|
jobs are kept disabled by default (IsActive = 0) and are only activated
|
|
when needed.
|
|
|
|
------------------------------------------------------------------------------
|
|
**Step** **Action** **Script / Procedure Used** **Purpose**
|
|
---------- ------------- ---------------------------------- ------------------
|
|
1 Review SELECT \* FROM JobControl WHERE Identify jobs to
|
|
monthly job Frequency=\'MONTHLY\'; be run this month.
|
|
list
|
|
|
|
2 Update run Edit ActivateMonthlyJob.sql Define monthly
|
|
date & time execution window.
|
|
|
|
3 Schedule & Run ActivateMonthlyJob.sql Create one-time
|
|
activate jobs SQL Agent
|
|
schedules.
|
|
|
|
4 Monitor Use JobExecutionHistory table or Track progress and
|
|
execution SQL Agent history outcomes.
|
|
|
|
5 Deactivate Run DeactivateMonthlyJob.sql Reset jobs to
|
|
jobs post-run inactive state.
|
|
------------------------------------------------------------------------------
|
|
|
|
**4.3.1 Default State: Jobs Disabled**
|
|
|
|
- All monthly jobs are initialized in the JobControl table with
|
|
IsActive = 0.
|
|
|
|
- This ensures that they do not run automatically unless explicitly
|
|
scheduled.
|
|
|
|
- Frequency is stored as \'MONTHLY\' for identification, but
|
|
scheduling depends entirely on the stored procedure.
|
|
|
|
Example Query:
|
|
|
|
SELECT JobName, IsActive, Frequency
|
|
|
|
FROM dbo.JobControl
|
|
|
|
WHERE Frequency = \'MONTHLY\';
|
|
|
|
**4.3.2 Activating and Scheduling Monthly Jobs**
|
|
|
|
To schedule monthly jobs for the upcoming month (e.g., October or
|
|
November), you'll use the ActivateMonthlyJob.sql script.
|
|
|
|
- Updates job activation flags (IsActive = 1).
|
|
|
|
- Calls the stored procedure usp_ScheduleMonthlyJob to dynamically
|
|
create or refresh SQL Agent schedules.
|
|
|
|
- Removes any old schedules and replaces them with new, one-time
|
|
schedules for the provided date and time.
|
|
|
|
> Example Query:
|
|
>
|
|
> EXEC dbo.usp_ScheduleMonthlyJob
|
|
>
|
|
> \@JobName = \'DEVL_MONTHLY_DTSCHGVB\',
|
|
>
|
|
> \@Env = \'DEVL\',
|
|
>
|
|
> \@RunDate = \'2025-10-31\',
|
|
>
|
|
> \@RunTime = \'19:26:00\';
|
|
|
|
What it does internally:
|
|
|
|
- Identifies the correct environment database:
|
|
|
|
- DEVL → DevlDUTASJobSchedule
|
|
|
|
- TEST → TestDUTASJobSchedule
|
|
|
|
- PROD → ProdDUTASJobSchedule
|
|
|
|
- Sets the job as active in JobControl.
|
|
|
|
- Detaches and deletes any existing schedule for the job.
|
|
|
|
- Creates a **new one-time SQL Agent schedule** for the given
|
|
date/time.
|
|
|
|
- Reattaches the schedule to the job.
|
|
|
|
- Prints confirmation once the job is scheduled.
|
|
|
|
> **4.3.3 Post-Run Deactivation**
|
|
>
|
|
> After monthly jobs finish executing successfully:
|
|
|
|
- Deactivate them again to prevent accidental re-runs.
|
|
|
|
- This is done using the DeactivateMonthlyJob.sql script.
|
|
|
|
Example Query:
|
|
|
|
UPDATE dbo.JobControl
|
|
|
|
SET IsActive = 0
|
|
|
|
WHERE Frequency = \'MONTHLY\';
|