HomeMy WebLinkAbout20040172.tiff The "COMPASS" Project
A Compliance Assistance Pilot Project
/OF•COtO
ur OI
* 'rS �c f,o, *
' J876%"
Colorado Department
of Public Health
and Environment
FINAL REPORT
July 2003
By:
it
Tetra Tech EM Inc.
1099 18th Street, Suite 1900 EXHIBIT
Denver, Colorado 80202 I i
t*
2004-0172
CONTENTS
Section page
1.0 INTRODUCTION 1-1
2.0 ASPHALT MANUFACTURING PLANTS 2-1
2.1 PROJECT DESCRIPTION 2-1
2.2 PROJECT IMPLEMENTATION 2-2
2.3 DATA ANALYSIS PROCEDURE 2-8
2.4 DATA ANALYSIS RESULTS 2-12
2.4.1 Air Quality Results 2-13
2.4.2 Storm Water Results 2-14
2.4.3 Hazardous Waste Results 2-15
2.4.4 Pollution Prevention Results 2-15
2.5 SUMMARY OF FINDINGS 2-16
3.0 HAZARDOUS WASTE SMALL QUANTITY GENERATORS 3-1
31 PROJECT DESCRIPTION 3-2
3.2 PROJECT IMPLEMENTATION APPROACH 3-2
3.3 DATA ANALYSIS PROCEDURE 3-6
3.4 DATA ANALYSIS RESULTS 3-8
3.5 SUMMARY OF FINDINGS 3-13
3.5.1 Statistical Analysis of Compliance Data 3-13
3.5.2 Descriptive Analysis 3-14
4.0 CHROME PLATING FACILITIES 4-1
4.1 PROJECT DESCRIPTION 4-1
4.2 PROJECT IMPLEMENTATION APPROACH 4-1
4.3 DATA ANALYSIS PROCEDURE 4-4
4.4 DATA ANALYSIS RESULTS 4-6
4.5 SUMMARY OF FINDINGS 4-10
5.0 SURFACE WATER TREATMENT PLANTS 5-1
5.1 PROJECT DESCRIPTION 5-1
5.2 PROJECT IMPLEMENTATION 5-2
5.3 SUMMARY OF FINDINGS 5-5
Final Report i Compass Project
!- CONTENTS(Continued)
APPENDICES
Appendix
APPENDIX A—ASPHALT PLANT COMPASS PROJECT
Appendix A-I —Example Anonymous Baseline Assessment Report
Appendix A-2—Compass Asphalt Plant Multi-Media Compliance and Pollution Prevention
Training Agenda
Appendix A-3—Results for Tests of Individual Indicators
Appendix A-4—Match between Asphalt Inspection Database Indicator Numbers and the
Questions They Represent
APPENDIX B—SQG COMPASS PROJECT
Appendix B-1 —SQG Data Collection Sheets
Appendix B-2—Raw SQG Compass Data
Appendix B-3 -SQG Compass Project Statistical Results
Appendix B-4-SQG Compass Project Graphical Results
APPENDIX C—CHROME PLATING COMPASS PROJECT
Appendix C-1 —Chrome Plating Compass Project Statistical Results
Appendix C-2—Match between Indicator Numbers and the Questions They Represent for the 28
Questions Common to all 4 Chrome Plating Facility Inspection Forms
Final Report ii Compass Project
CONTENTS(Continued)
TABLES
Table
1-1 COMPASS SECTOR SUMMARY
2-1 STATISTICAL ANALYSIS SUMMARY FOR INDIVIDUAL AIR QUALITY COMPLIANCE
INDICATORS
2-2 STATISTICAL ANALYSIS SUMMARY FOR INDIVIDUAL STORM WATER
COMPLIANCE INDICATORS
2-3 STATISTICAL ANALYSIS SUMMARY FOR INDIVIDUAL HAZARDOUS WASTE
COMPLIANCE INDICATORS
2-4 STATISTICAL ANALYSIS SUMMARY FOR INDIVIDUAL POLLUTION PREVENTION
INDICATORS
2-5 INDICATOR CATEGORY ANALYSIS RESULTS(QUESTION 2)
2-6 INDICATOR CATEGORY ANALYSIS RESULTS(QUESTIONS 3 and 4)
2-7 AVERAGE COMPLIANCE SCORES
3-1 SQG GROUP ASSIGNMENTS
3-2 COMPARISON OF THE AVERAGE COMPLIANCE RATES PER FACILITY AMONG ALL
GROUPS
3-3 PROPORTION OF COMPLIANT FACILITIES AMONG GROUPS
3-4 SUMMARY OF PERCENT"YES"RESPONSES TO GENERAL QUESTIONS
3-5 SQG QUALITATIVE DATA SUMMARY
3-6 GROUP 3 ONLY QUESTION RESPONSES
3-7 SELECT COVERAGE AREA PERFORMANCE SUMMARY
3-8 GRAPHICALLY NOTICEABLE COMPLIANCE IMPROVEMENT BY COVERAGE AREA
4-1 CHROME PLATER GROUP ASSIGNMENTS
4-2 COMPARISON OF THE AVERAGE COMPLIANCE RATES PER FACILITY AMONG
GROUPS
4-3 AVERAGE PERCENT COMPLIANCE AMONG VIOLATION CATEGORIES
5-1 PLF RESULTS SUMMARY FOR SWTPs
Final Report iii Compass Project
CONTENTS(Continued)
FIGURES
Figure
2-1 ASPHALT PLANT COMPASS PROJECT TIMELINE
2-2 ASPHALT PLANT COMPASS PROJECT MAP
2-3 ASPHALT PLANT DATABASE DATA INPUT SCREEN
2-4 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE AGGREGATE AIR QUALITY
ANALYSIS FOR CRITICAL COMPLIANCE INDICATORS
2-5 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE INDIVIDUAL INDICATOR AIR
QUALITY ANALYSIS FOR CRITICAL COMPLIANCE INDICATORS
2-6 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE AGGREGATE AIR QUALITY
ANALYSIS FOR NON-CRITICAL COMPLIANCE INDICATORS
2-7 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE INDIVIDUAL INDICATOR AIR
QUALITY ANALYSIS FOR NON-CRITICAL COMPLIANCE INDICATORS
2-8 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE AGGREGATE AIR QUALITY
ANALYSIS FOR PLANTS
2-9A BASELINE TO FOLLOW-UP COMPLIANCE CHANGE INDIVIDUAL PLANT AIR
QUALITY ANALYSIS
2-9B BASELINE TO FOLLOW-UP COMPLIANCE CHANGE INDIVIDUAL PLANT AIR
QUALITY ANALYSIS
2-10 BASELINE TO FOLLOW-UP DISTRIBUTION OF AGGREGATE AIR QUALITY
COMPLIANCE AMONG ALL PLANTS
2-11 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE AGGREGATE STORM WATER
ANALYSIS FOR CRITICAL COMPLIANCE INDICATORS
2-12 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE INDIVIDUAL INDICATOR STORM
WATER ANALYSIS FOR CRITICAL COMPLIANCE INDICATORS
2-13 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE AGGREGATE STORM WATER
ANALYSIS FOR NON-CRITICAL COMPLIANCE INDICATORS
2-14A BASELINE TO FOLLOW-UP COMPLIANCE CHANGE INDIVIDUAL INDICATOR STORM
WATER ANALYSIS FOR NON-CRITICAL COMPLIANCE INDICATORS
2-I4B BASELINE TO FOLLOW-UP COMPLIANCE CHANGE INDIVIDUAL INDICATOR STORM
WATER ANALYSIS FOR NON-CRITICAL COMPLIANCE INDICATORS
2-15 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE AGGREGATE STORM WATER
ANALYSIS FOR ALL PLANTS
2-16 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE INDIVIDUAL PLANT STORM
WATER ANALYSIS
2-17 BASELINE TO FOLLOW-UP DISTRIBUTION OF AGGREGATE STORM WATER
COMPLIANCE AMONG ALL PLANTS
2-18 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE AGGREGATE HAZARDOUS
WASTE ANALYSIS FOR CRITICAL COMPLIANCE INDICATORS
2-19 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE INDIVIDUAL INDICATOR
HAZARDOUS WASTE ANALYSIS FOR CRITICAL COMPLIANCE INDICATORS
2-20 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE AGGREGATE HAZARDOUS
WASTE ANALYSIS FOR NON-CRITICAL COMPLIANCE INDICATORS
2-21 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE INDIVIDUAL INDICATOR
HAZARDOUS WASTE ANALYSIS FOR NON-CRITICAL COMPLIANCE INDICATORS
2-22 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE AGGREGATE HAZARDOUS
WASTE ANALYSIS FOR ALL PLANTS
Final Report iv Compass Project
2-23 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE INDIVIDUAL PLANT
HAZARDOUS WASTE ANALYSIS
2-24 BASELINE TO FOLLOW-UP DISTRIBUTION OF AGGREGATE HAZARDOUS WASTE
AMONG ALL PLANTS
2-25 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE AGGREGATE POLLUTION
PREVENTION ANALYSIS FOR CRITICAL COMPLIANCE INDICATORS
2-26 BASELINE TO FOLLOW-UP COMPLIANCE RATE CHANGE INDIVIDUAL INDICATOR
POLLUTION PREVENTION ANALYSIS FOR CRITICAL COMPLIANCE INDICATORS
2-27 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE AGGREGATE POLLUTION
PREVENTION ANALYSIS FOR NON-CRITICAL COMPLIANCE INDICATORS
2-28A BASELINE TO FOLLOW-UP COMPLIANCE CHANGE INDIVIDUAL INDICATOR
POLLUTION PREVENTION ANALYSIS FOR NON-CRITICAL COMPLIANCE
INDICATORS(FIRST HALF)
2-28B BASELINE TO FOLLOW-UP COMPLIANCE CHANGE INDIVIDUAL INDICATOR
POLLUTION PREVENTION ANALYSIS FOR NON-CRITICAL COMPLIANCE
INDICATORS(SECOND HALF)
2-29 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE AGGREGATE POLLUTION
PREVENTION ANALYSIS FOR ALL PLANTS
2-30 BASELINE TO FOLLOW-UP COMPLIANCE CHANGE INDIVIDUAL PLANT POLLUTION
PREVENTION ANALYSIS
2-31 BASELINE TO FOLLOW-UP DISTRIBUTION OF AGGREGATE POLLUTION
PREVENTION AMONG ALL PLANTS
3-1 GROUP COMPLIANCE COMPARISON FOR COVERAGE AREA"PTG"
(Improperly Closed Containers of Hazardous Waste)
3-2 COMPLIANCE ASSURANCE PROGRAM COMPARISON WITH COMPLIANCE
3-3 DESIGNATED ENVIRONMENTAL COORDINATOR COMPARISON WITH COMPLIANCE
4-1 CHROME PLATING COMPASS PROJECT TIMELINE
4-2 CHROME PLATING COMPASS PROJECT MAP
4-3 CHROME PLATING COMPLIANCE PERFORMANCE FOR REPORTING INDICATORS
4-4 CHROME PLATING COMPLIANCE PERFORMANCE FOR OPERATION AND
MAINTENANCE PLAN INDICATORS
4-5 CHROME PLATING COMPLIANCE PERFORMANCE FOR RECORD KEEPING
INDICATORS
4-6 CHROME PLATING COMPLIANCE PERFORMANCE FOR INITIAL NOTIFICATION
INDICATORS
4-7 CHROME PLATING COMPLIANCE PERFORMANCE FOR NEW TANK INSTALLATION
4-8 CHROME PLATING COMPLIANCE PERFORMANCE FOR NOTIFICATION OF
COMPLIANCE STATUS INDICATORS
5-1 CPE COMPASS PROJECT TIMELINE
5-2 CPE COMPASS PROJECT MAP
5-3 NUMBER OF LEVEL A AND B PLFs PER SWTP
5-4 COMPARISON OF LEVEL A AND B PLF OCCURRENCES
5-5 COMPARISON LEVEL A AND B PLF"ACTION AND NO ACTION TAKEN" RATES
5-6 LEVEL A"ACTION AND NO ACTION TAKEN" SUMMARY
5-7 LEVEL B"ACTION AND NO ACTION TAKEN" SUMMARY
Final Report v Compass Project
1.0 INTRODUCTION
In 2000,the U.S. Environmental Protection Agency (EPA)encouraged state regulatory agencies to
integrate quantitative outcome-based performance measures into enforcement and compliance assurance
programs. Under the Colorado Environmental Performance Partnership Agreement 2001-2002,the
Colorado Department of Public Health and Environment(CDPHE)and EPA initiated the Compliance
Assurance Pilot Project,or"Compass Project." The Compass Project had two main purposes. First,the
air,water,and hazardous waste divisions of CDPHE wanted to test the affect of compliance assistance
methods and measure the impact on compliance rates. Second,CDPHE wanted to develop an electronic
query and reporting capability that could measure the environmental results of compliance and
enforcement activities across all media. This report,prepared by Tetra Tech,summarizes the results of
the air,water,and hazardous waste Compass projects. The results of the electronic reporting project are
not expected until October 2003.
The Compass pilot projects involved four sectors of regulated facilities. Table 1-1 summarizes the
sectors,CDPHE divisions in charge of each Compass sector project,reasons the sector was included in
the Compass project,number of facilities in that sector in Colorado,and number of facilities in the
Compass project.
r
Compass project designs varied among each of the four sectors because the existing compliance and
enforcement programs in each sector were inherently more conducive to different designs. The Compass
project designs for each of the four sectors are summarized below.
Asphalt Manufacturing:
• Conduct multi-media compliance and pollution prevention inspections to establish a
compliance baseline for participating asphalt plants.
• Provide compliance and pollution prevention assistance to asphalt plants.
• Conduct follow-up multi-media inspections to gather compliance and pollution prevention
data from participating asphalt plants.
• Measure change in compliance and pollution prevention indicators from baseline to follow-up
inspections.
Final Report 1-1 Compass Project
TABLE 1-1
COMPASS SECTOR SUMMARY
Fadlilies in Facilities in
Compass Sector CDPHE Division Reasons for Inclusion in Compass Colorado Compass
Asphalt Air Pollution • Unsatisfactory compliance 50 44
manufacturing Control Division performance
plants (APCD) • Potential for negative impact on
human health and environment
• APCD available resources
• Industry willingness
Small quantity Hazardous • Historically low number of —800 49
hazardous waste Materials and inspections
generators(SQG) Waste • HMWMD had good
Management understanding of traditional
Division compliance rates
(IIMWMD)
Chrome platers APCD • Unsatisfactory compliance 29 29
performance
• Potential for negative impact on
human health and environment
• Consistent equipment and
technology within sector
Surface water Water Quality • Increasing difficulty for SWTPs —200 28
treatment plants Control Division to comply with supply treatment
(WQCD) process requirements
• Potential for negative impact on
human health and environment
• Need for consistent particle
removal
SQGs and Chrome Platers:
• Divide sector(s)into three groups:
I. Facilities that received inspections in a conventional manner with no advance notice
(control group,group I).
2. Facilities that received a letter indicating that an inspection would occur at the facility
within 6 months(group 2).
3. Facilities that received the same letter as group 2 facilities,as well as extensive
written and verbal compliance assistance and pollution prevention information
(group 3).
• Conduct group I,2,and 3 inspections
• Evaluate differences in compliance among groups 1, 2,and 3.
Surface Water Treatment Plants:
• Develop performance criteria.
• Select treatment plants at which to apply performance criteria.
• Visit treatment plants and apply performance criteria.
• Identify performance deficiencies and send information to treatment plants.
• Follow up with treatment plants to evaluate whether and how they addressed performance
deficiencies.
• Measure percent of treatment plants that addressed deficiencies.
Final Report 1-2 Compass Project
The Compass pilot program for each sector is explained in greater detail in the subsequent sections of this
report. Each section contains the following information:
• Project description
• Project implementation
• Data analysis procedures
• Data analysis results
• Summary of findings
Analysis of Compass data employed both descriptive(graphical methods)and quantitative(formal
statistical tests)approaches. Graphical presentations of the data were used as a preliminary step to
provide a quick visual assessment of the relationship among the various parameters of each study(for
example,facility responses,relationship among performance indicators,etc.). Graphic presentations are
also a relatively efficient means of summarizing large amounts of information for many variables.
Formal statistical methods were used,to the extent practicable,in cases where it was desired to make
more focused and conclusive statements concerning relationships among facility responses and to test
specific hypotheses. The approach for implementing one or more statistical tests was the same for each of
the Compass studies. First, questions of general interest were formulated for each study. Each question
posed was related to a particular study objective,such as comparing rates of compliance between
inspection years or groups of facilities. Each question was then rephrased in the form of null(H0)and
alternative(HA)hypotheses statements. The hypothesis statements present the study questions in a form
that is suitable for testing using one or more specific statistical methods.
For three of the four Compass projects,formal statistical methods were used to analyze the differences
between groups(baseline and follow up for asphalt manufacturing plants; groups 1, 2,and 3 for SQGs
and chrome plating facilities). All formal statistical analyses were completed using a nominal confidence
level of 95 percent. This means that one can conclude with at least 95 percent confidence that the results
of the statistical tests cannot be attributed to chance alone. The specific data analysis procedure for each
sector is described in the respective"data analysis procedures"section. An overview of the statistical
questions for asphalt plants, SQGs,and Chrome platers follows.
Asphalt Manufacturing:
For the asphalt plant Compass project,the statistical analysis was designed to four of questions:
1. For each assessment checklist question(indicator),was there a statistically significant change in
the proportion of compliant plants from the first year to the second year?
Final Report 1-3 Compass Project
2. For each plant and indicator category(air quality,hazardous waste,wastewater,and pollution
prevention)was there a statistically significant change in the proportion of indicators showing
compliance from the first year to the second year?
3. For each indicator category,was there a statistically significant difference in the number of
indicators that showing increase versus decrease from the first year to the second year?
4. For each indicator category,was there a statistically significant difference in the magnitude of
change(across all indicators in the category)in the proportion of compliant plants between the
first year and the second year?
The specific statistical tests used to analyze each of the asphalt statistical questions are explained in
Section 2.3 and results are presented in section 2.4.
SQGs and Chrome Platers:
The Compass project design was similar for SQGs and chrome platers. Therefore, the statistical analyses
for the SQGs and chrome platers was designed to ask the same two questions:
1. Is there a difference in the average compliance rotes per facility among the three groups(group 1,
group 2,and group 3)?
2. Is there an overall difference in compliance among the three groups(group 1, group 2,and group
3)with respect to individual performance indicators(or coverage areas)and for all indicators(or
coverage areas)combined?
The specific statistical tests used to analyze each of the SQG statistical questions are explained in Section
3.3 and results are presented in Section 3.4;for chrome platen,statistical questions are explained in
Section 4.3 and results are presented in Section 4.4.
Surface Water Treatment Plants:
The Water Quality Control Division(WQCD)conducted comprehensive performance evaluations(CPE)
at 28 surface water treatment plants(SWTP)to identify performance limiting factors(PLF)that can
negatively affect the performance of the SWTP,and thus the treated water quality. WQCD did not
conduct a statistical analysis of the data it collected because baseline data or separate years of data were
not available to use for comparison. WQCD telephoned each of the SWTPs about 2 years after the CPE
to gather information regarding the SWTPs actions to mitigate the PLFs. The results of the program,for
the purpose of the Compass project,were evaluated by analyzing the number and type of PLFs identified
and the percent of the facilities that took action to eliminate those PLFs. A summary of findings for
SWTPs is presented in Section 5.3.
,rr
Final Report 1-4 Compass Project
2.0 ASPHALT MANUFACTURING PLANTS
RESULTS AT A GLANCE
The asphalt plant Compass project resulted in statistically meaningful improvements in
compliance by the asphalt manufacturing plants involved in the project from 2001(baseline)to
2002 (follow-up). The statistical analysis of baseline versus follow-up results suggest the
following conclusions about the asphalt plant Compass project:
✓ Overall compliance of asphalt plants significantly improved. 109 indicators(75
percent)increased in performance.
✓ Thirty-four indicators increased in a statistically significant manner.
✓ Thirty indicators(25 percent)decreased in performance.
✓ Four indicators decreased in a statistically significant manner;none were compliance
related, all were pollution prevention indicators.
✓ For all indicator groups(air quality,storm water,hazardous waste,pollution
prevention,and all indicators [see text for further explanation of indicator categories])
there was a statistically significant increase from 2001 to 2002 in both the number of
indicators showing compliance and the magnitude of change of compliance(see
Section 2.3 for further statistical analysis description).
✓ The average compliance scores in the indicator categories increased in the range of 2
to 31 percent.
✓ Twenty-nine critical compliance indicators(CCI)were identified;9 CCIs showed
statistically significant improvement and no CCIs decreased in performance.
See Section 2.2,Task 6 for explanation of CCIs.
2.1 PROJECT DESCRIPTION
The Air Pollution Control Division's(APCD's)Compass project involved the asphalt manufacturing
industry. The project was implemented through a partnership between APCD and the Colorado Asphalt
Pavement Association(CAPA). Historically,APCD relied on the conventional approach of assessment
and enforcement. The conventional approach uses routine enforcement assessments to evaluate
compliance with CDPHE regulations followed by enforcement actions such as compliance orders,notices
of violation,and consent orders to correct noncompliance conditions observed. Enforcement actions
efforts typically result in levying fines as a penalty for noncompliance.
Final Report 2-1 Asphalt Plant Compass Project
In contrast,the asphalt plant Compass project provides compliance and pollution prevention assistance to
participating industry members in a manner that reduces the use of enforcement action and penalties.
Compass project asphalt plants were taken off routine enforcement assessment schedules for 2 years.
Instead they received a site visit each year in which APCD assessed air,stone water,and hazardous waste
compliance as well as pollution prevention. Plants participating in the project were still subject to
enforcement actions if citizen complaints occurred or an inspector driving by a plant observed an obvious
non-compliance situation. This project was designed to test the effectiveness of compliance assistance,
rather than enforcement actions,as a means of achieving better compliance rates among asphalt plants.
APCD selected the asphalt manufacturing sector for four reasons,as follows:
1. Unsatisfactory Compliance Performance: In the past, APCD perceived the overall compliance
of asphalt plants to be unsatisfactory.
2. Potential for Negative Affect on Human Health and Environment: Colorado is experiencing
an increasing growth pattern. Over the last five years Colorado's population has increased by
more than 600,000',with an average of more than 31,000 new building permits per year.'
Highway construction in Colorado includes 34 major projects with an estimated cost of over 450
million dollars.' Asphalt production in Colorado increased from eight million tons in 1995 to ten
million tons in 2000,and nearly twelve million tons are projected for 2003.4 This population
growth and associated highway construction will be distributed throughout the state and have the
potential for adverse environmental and human health impacts.
3. Available Resources: Fiscal years 2001 and 2002 work plan budgets allowed APCD to allocate
resources to develop a partnership with the asphalt manufacturing industry.
4. Industry Willingness: When CDPHE approached the asphalt manufacturing industry they
agreed to partner with CDPHE on the Compass project. Forty four facilities operated by CAPA
members chose to participate in the project and CAPA agreed to take a lead role in organizing
and working with the partner companies.
2.2 PROJECT IMPLEMENTATION
An overview of the asphalt plant Compass project milestones and schedule are presented in Figure 2-1
and discussed in further detail in the sections following the timeline.
Colorado Division of Local Government and http://cfapp.rockymountainnews.corn/census2000/coloradoGrowth.cfm
2 U.S.Census Bureau
r Colorado Department of Transportation
°Phone conversation with Tom Peterson,Executive Director,CAPA
Final Report 2-2 Asphalt Plant Compass Project
FIGURE 2-1
ASPHALT PLANT COMPASS PROJECT TIMELINE
Task 1:Develop Task 3:Develop Task 5:Create and Task 7:Provide compliance Task 9:Send follow-
Compass project assessment populate assessment assistance and write Mid-Term up assessment reports
concept and interest checklists(4/01) database(10/01) Industry Report(3/02) to plants(4/03)
(9/00)
•
{ Jr ♦ 1
Task 2:Propose Task 4: Conduct Task 6:Send Baseline Task 8:Conduct
program to Colorado baseline year Task 10:Write
asphalt plants(12/00) assessments(5/01- Assessment Reports[o follow-up final report
9/0I) plants(2/02) assessments
(5/02 to 10/02) (3/03)
Task 1: Develop Compass Project Concept and Interest
During the asphalt plant Compass project,APCD worked with asphalt plants to develop compliance
assistance tools,a general permit and emission model to streamline permitting,and joint training on
multi-media compliance assistance,pollution prevention opportunities,and new technologies. APCD's
goal was to partner with CAPA and the Colorado Department of Transportation to see if such a
compliance assistance partnership would result in quantifiable improvements in compliance and
environmental outcome-based performance measures.
Task 2: Propose Program to Colorado Asphalt Plants
About 50 asphalt plants operated in Colorado in 2001.5 APCD,in cooperation with CAPA,contacted
each of these asphalt plants. CDPHE offered the following incentives for participation in the project:
1. No scheduled enforcement assessments in any media for 2 years(APCD still responded to citizen
complaints and observations by inspectors in the normal course of his/her other duties)
2. APCD and Pollution Prevention staff time for technical and compliance assistance
3. Opportunity to correct any compliance issues discovered by APCD,without enforcement action
from CDPHE(unless identified by a citizen complaint,by an APCD inspector in the normal
course of his/her other duties,or during an enforcement assessment after the project)
In exchange,the plants that agreed to participate in the project were required to dedicate staff time to
answer multimedia compliance and pollution prevention questions and allow access to the plant for the
baseline and follow-up on-site assessments for 2 years. CDPHE also expected the plants would remedy
any noncompliance discovered,but did not require it.
5 According to CAPA,50 asphalt plants includes all non-portable,industry plants;that is,plants with a primary business of
asphalt manufacturing. This excludes,for example,any municipality that manufactures asphalt.
Final Report 2-3 Asphalt Plant Compass Project
In 2001,44 plants(more than 85 percent of Colorado's asphalt plants)representing more than 90 percent
of the asphalt generated in Colorado(according to CAPA)agreed to participate in the program. The
remaining 10 percent of the asphalt produced in Colorado was manufactured by relatively small
operations including either portable asphalt plants or municipalities with road paving divisions. The
plants involved in the asphalt plant Compass project are shown in Figure 2-2.
Task 3: Develop Assessment Checklists
APCD developed several compliance and pollution prevention checklists relevant to asphalt plants.
Checklists were developed for:
• Pre-assessment and pre-visit
• Air quality regulations(general,baghouse,scrubber,and fugitive dust)
• Hazardous waste regulations
• Stonnwater regulations
• Pollution prevention opportunities
Task 4: Conduct Baseline Year Assessments
APCD conducted a baseline assessment during the 2001 production season(5/01 to 9/01)at 44 asphalt
plants using the multimedia compliance checklists. During this assessment,one or two CDPHE staff
visited each plant to conduct air quality,hazardous waste, storm water compliance assessments,as well as
to assess the extent to which the plants had incorporated pollution prevention techniques and technologies
into the operations. Each assessment lasted about 4 hours and required one or two asphalt plant staff to
answer more than 300 compliance and pollution prevention questions. At the end of each assessment,
CDPHE conducted an exit interview with plant or company personnel to provide a general overview of
the results of the assessment and discuss critical compliance issues and significant pollution prevention
opportunities.
Task 5: Create and Populate Assessment Database
APCD created a database to receive and manipulate the asphalt plant compliance and pollution prevention
checklist information through a user-friendly graphical interface. An example data input screen is shown
in Figure 2-3. The database manipulates the data,calculates basic statistical information, and generates a
report for each plant that summarizes its compliance and pollution prevention status,issues,and
recommendations to address any issues. APCD populated the assessment database with the data it
collected in the baseline year assessment(and later in the follow-up assessment,see Task 8).
Final Report 2-4 Asphalt Plant Compass Project
Ks
��� Sedgwlck
Logan
/
::II1I
�_r 1.• Feld�` r�, :A_ Z I
1
_ —
Merges
Creed Her e+
rZ Blanco r • V „i!ngto
1 n Yurna
— - -
n D�rver • Art,,,,,,
I GCs!fltild • PH qra pa hi,_
® Eagle; fl ..1
:-L)..1'1'1,:)-(r
ii^�" pot I,,icy
f / Kit Carson
< r /I i 'bert �.—
✓1` P�In Lyka ru,?I�as 1
11
Park lT._a i Mesa Helm THE-�- t P„�;..
o ,_ n„��.an,
w rA 1
<----
i,i rrls;n i L\ ChaN r -
„_
I I,stmee _ �-- runt 1 Rlo,vn
` _ N C rrn:I
1 —T
(Jura;. — 11 PuehH ���
ti aguacFie isW_r I
San ',husk N HreSale
Sent �P Pme,.rs
\--,,,__ �. —� Otero
es
J alore� ���' ,-r Jr�.an r ,_
M aral .elan%
:e C 'spry Alamos' ''� -� 9
M mrersna V ____-( a,Ar,ma, Hasa
La Plain Areh dua Crunolw
I 1
/ C. I
0 25 50 100 150 200
r
Legend Miles
�� Asphalt Manufacturing Location N
Interstate Highway FIGURE 2-2
County Boundary
ASPHALT MANUFACTURING
COMPASS PROJECT
FIGURE 2-3 ASPHALT PLANT DATABASE DATA INPUT SCREEN
r:I
Age Olt Now swtt Apia eeoed<slob NNW u►
el-0 l^'iet_j!{I1;is!`_;N•►• sc;ei' tB A. C4-1
ns Sant see —— 0 ta IRS. --•;e-
Saki A Om breenOon rasiDenletel
Falb or FesiEse� Comments about lnWnciioa-i.R.ptul
Falb R.r�
Soled a.dt.dd.b mod in this inspection kr.ddies that i.s4 aunt w wood out)---
'ff.L.S.c�AS.^f^!+' P P i.«:r.,. P �' :I':Npr A: P
P San r P
.4 Ern:ron, P Err.ro?_rr' P P2ARsdb.. r Rermre arpra si dedofebi
Add N.now dteddi.b sslscled.bwe b Me database
-- -- -- Comments about Ins Wien-Hot is Report
Hi eat derlitle Mesa
elm Sere selected l ins
ddtediete lYdeeed to be AtIdassIiis I
iroled.d M Mb brsellse
( InnsP a
caed'h8l PrM.eres Pmensit) 14, eaphos••I Scrubber( ro.episs I Peptises I NW ClIalpl P2 A ; P21 ;Elm.ntl.tI
to go to:
OTHER FUNCTIONS
Add.tow l.aiilf I Wed et inp.dboe to go lac
REPORTS
Ste.._,Rope.
lirdte.dUtO/9E Its lobe l.►ENlet9tM ee21NI
restim, f Wig --
Task 6: Send Baseline Assessment Reports to Plants
After the baseline assessments, APCD entered the assessment data into the asphalt manufacturing
database. The database automatically generated a comprehensive report(Baseline Assessment Report)
that identified compliance issues observed during baseline assessment for each plant. The Baseline
Assessment Report addressed multimedia compliance issues. The report also identified pollution
prevention opportunities. Within each media,compliance issues were divided into"critical compliance
indicators" (CCI)and"noncritical compliance indicators." CDPHE media(air, storm water,hazardous
waste,and pollution prevention)representatives determined the CCI for each media as follows:
■ Air CCIs: checklist questions likely to result in an enforcement action if discovered during a
compliance assessment.
• Storm Water CCIs: checklist questions that addressed the presence of an adequate Storm
Water Management Plan and the adequacy of implementation of major elements of this plan.
• Hazardous Waste CCIs: checklist questions most directly related to a potential for release
to the environment or likely to result in an enforcement action if discovered during a
compliance assessment.
Final Report 2-6 Asphalt Plant Compass Project
• Pollution Prevention CCIs:checklist questions that addressed pollution prevention
opportunities or practices most likely to significantly improve environmental performance
(and often generate significant cost savings).
The Baseline Assessment Report provides compliance rates and recommendations for improving the
compliance rate for each CCI. Appendix A-1 contains an example Baseline Assessment Report for an
anonymous plant.
Task 7: Provide Compliance Assistance and Write Mid-Term Industry Report
After the first assessment,APCD assisted the Compass asphalt plants with compliance improvement in all
media and pollution prevention for plant-wide operations. Specifically,APCD assistance included:
• Exit interview in which CDPHE discussed critical compliance issues and significant pollution
prevention opportunities with the plant representatives.
• Multi-media compliance and pollution prevention training;a I-day training for project
participants held on March 27, 2002,see Appendix A-2 for agenda. The training covered
the following regulatory topics: air pollution control, stormwater,hazardous waste,fuel
containment, spill prevention,control,and countermeasures,and Tier II chemical
inventory reporting. Other presentations in the training covered the Colorado
Environmental Leadership,EPA Performance Track,Environmental Management
Systems,new technologies and best practices to save fuel and improve environmental
performance,and the National Asphalt Pavement Association perspectives on
environmental issues.
• One-on-one conferences with APCD and/or Pollution Prevention staff were available to
project participants to answer questions or explain Baseline Assessment Report
recommendations.
In March 2002,APCD wrote a Mid-Term Industry Report(www.cdphe.state.co.us/ap/down
/capaasphalt.pdf)summarizing the findings of the baseline assessments for all Compass plants. The mid-
term report summarized compliance issues and pollution prevention opportunities identified in the
Baseline Assessment Reports for all 44 plants.
After each Compass asphalt plant received: (1)an exit interview addressing critical compliance issues and
significant pollution prevention opportunities, (2)an individual plant Baseline Assessment Report,(3)a
1-day multi-media compliance assistance and pollution prevention training,(4)an opportunity for one-on-
one conferences with CDPHE project staff;and(5)the Mid-Term Industry Report;the asphalt plants had
the opportunity to improve their compliance rate and pollution prevention implementation.
Task 8: Conduct Follow-Up Assessments
Final Report 2-7 Asphalt Plant Compass Project
APCD conducted a follow-up assessment of 42 of the original 44 plants during the 2002 production
season (5/02 to 11/02). Two plants did not participate in the second year of assessments because one
closed and a mobile plant did not return to Colorado. During the follow-up assessment,plants were asked
the same compliance and pollution prevention questions as during the baseline assessment.
Task 9: Send Follow-Up Facility Reports to Plants
After the follow-up assessments,APCD entered the assessment findings into the asphalt plant Compass
project database. The database generated reports similar to the Baseline Assessment Report(see Task 6)
using follow-up assessment information. APCD sent each of the 42 plants a second plant-specific report
addressing multimedia compliance issues and pollution prevention opportunities that APCD identified in
the follow-up assessments.
Task 10: Write Final Report
In March 2003, APCD wrote a final report(www.cdphe.state.co.us/ap/down/capareport.pd0 summarizing
the findings of the baseline and follow-up assessments for all of the Compass plants. The final report
summarized compliance issues and pollution prevention opportunities in the baseline and follow-up
facility reports for all plants and compared plant performance from baseline to follow-up assessment for
air quality,hazardous waste,and storm water compliance,as well as pollution prevention opportunities.
2.3 DATA ANALYSIS PROCEDURE
The APCD analyzed the baseline and follow-up asphalt plant data quantitatively, using a statistical
methodology,and descriptively,using charts and graphs to visually represent project results. This section
describes APCD's statistical data analysis procedure. The statistical and graphical results are presented in
Section 2.4. The statistical analysis for asphalt manufacturing plants addressed four questions that are
stated in this section. Each question is followed by a description of the specific statistical methods used
to evaluate that question.
Final Report 2-8 Asphalt Plant Compass Project
Question 1: For each compliance indicator.*was there a significant change in the proportion
of compliant plants between the baseline(2001)and the follow-up assessment
(2002)?
*Note:a compliance indicator is an assessment checklist question.
Statement of Null and Alternative Hypotheses:
Ho: The proportion of plants in compliance with respect to indicator X did not change from
2001 to 2002
H,: The proportion of compliant plants measured in 2002 is different than the proportion
measured in the baseline(2001)
The statistical approach used to address Question 1 is based on the principle that plants and assessment
year constitute a"matched pair." That is.the same plants are inspected during both years of the study:
therefore, the analysis must account for the dependent relationship between the proportion of compliant
plants reported during each year. The Question 1 statistical analysis can be performed using the binomial
test. The data are first arranged in a special 2 X 2 table, as shown below:
Response 2001 2001
YES NO
2002 A B
YES
2002 C D
NO
The four cells in the table are defined as follows:
A: Plant in compliance in 2001 and 2002
B: Plant out of compliance in 2001,but in compliance in 2002(showing increase in compliance)
C: Plant in compliance in 2001,but out of compliance in 2002(showing decrease in compliance)
1): Plant out of compliance in 2001 and 2002
If there is no change in the proportion of compliant plants between years,the number of plants reported in
cells R and C should be equal. Stated another way, if the number of plants reporting an increase in
compliance between years is equal to the number of plants reporting a decrease in compliance between
years.the net change is zero. The binomial test can be used to calculate the exact probability of observing
at least B or C number of cases by chance alone. That is,if the probability of an individual plant showing
either an increase or decrease in compliance is fixed at 0.50(50 percent),the number of cases that arc
expected in both cells B and C can be calculated if there was no net change between years. If a greater
number of plants is observed in cell B compared to the number that would he expected by chance alone,
Final Report 2-9 Asphalt Plant Compass Project
Question 1: For each compliance indicator,*was there a significant change in the proportion
of compliant plants between the baseline(2001)and the follow-up assessment
(2002)?
*Note: a compliance indicator is an assessment checklist question.
Statement of Null and Alternative Hypotheses:
Ho: The proportion of plants in compliance with respect to indicator X did not change from
2001 to 2002
HA: The proportion of compliant plants measured in 2002 is different than the proportion
measured in the baseline(2001)
The statistical approach used to address Question 1 is based on the principle that plants and assessment
year constitute a"matched pair." That is,the same plants are inspected during both years of the study;
therefore,the analysis must account for the dependent relationship between the proportion of compliant
plants reported during each year. The Question 1 statistical analysis can be performed using the binomial
test. The data are first arranged in a special 2 X 2 table, as shown below:
Response 2001 2001
y- YES NO
2002 A B
YES
2002 C D
NO
The four cells in the table are defined as follows:
A: Plant in compliance in 2001 and 2002
B: Plant out of compliance in 2001,but in compliance in 2002 (showing increase in compliance)
C: Plant in compliance in 2001,but out of compliance in 2002 (showing decrease in compliance)
D: Plant out of compliance in 2001 and 2002
If there is no change in the proportion of compliant plants between years,the number of plants reported in
cells B and C should be equal. Stated another way,if the number of plants reporting an increase in
compliance between years is equal to the number of plants reporting a decrease in compliance between
years,the net change is zero. The binomial test can be used to calculate the exact probability of observing
at least B or C number of cases by chance alone. That is,if the probability of an individual plant showing
either an increase or decrease in compliance is fixed at 0.50(50 percent),the number of cases that are
expected in both cells B and C can be calculated if there was no net change between years. If a greater
number of plants is observed in cell B compared to the number that would be expected by chance alone,
Final Report 2-9 Asphalt Plant Compass Project
APCD can conclude there was a statistically significant increase in compliance between years. If a
greater number of plants is observed in cell C compared to the number that would be expected by chance
alone,APCD can conclude there was a statistically significant decrease in compliance between years.
The results for tests conducted for individual indicators are provided in Section 2.4 and Appendix A-3,
Table A-3-1.
The statistical approach used to address Question 2 is the same as that used for Question 1. The only
difference is that Question 2 tests for a significant change in the proportion of indicators showing
compliance at individual plants. Each test is conducted for indicators pooled into four assessment
categories(air,hazardous waste,storm water,pollution prevention)and for a single category where all
indicators are combined. The analysis is explained in further detail below;results of these tests are
provided in Section 2.4 and Appendix A-3,Table A-3-2.
Question 2: For each plant and indicatory category,was there a significant change in the
proportion of indicators showing compliance in the baseline(2001)versus the
follow-up assessment(2002)?
Statement of Null and Alternative Hypotheses:
Each indicatory category consists of indicator questions pooled into four discrete assessment
categories(air,hazardous waste,storm water,pollution prevention)plus a single category
where all indicator questions are combined.
H0: The proportion of indicators in compliance with respect to plant X and indicator
category Y has not changed or has declined from 2001 to 2002.
HA: The proportion of indicators scored as compliant with respect to plant X and indicator
category Y in 2002 is greater than the proportion measured in the baseline(2001),
Questions 3 and 4 are discussed together, as they are closely related and rely on the same initial
manipulation of the data to conduct each analysis.
Final Report 2-10 Asphalt Plant Compass Project
Question 3: For each indicator category,is there a significant difference in the number of
indicators that show an increase versus a decrease in compliance between 2001 and
2002?
Statement of Null and Alternative Hypotheses:
Each indicator category consists of indicator questions pooled into four discrete assessment
categories(air,hazardous waste, storm water,pollution prevention)plus a single category where
all indicator questions are combined.
H0: The number of indicators in aggregate group X that showed an increase in compliance
between 2001 and 2002 is less than or equal to the number of indicators that showed a
decrease in compliance between the 2 years.
HA: The number of indicators in aggregate group X that showed an increase in compliance
between 2001 and 2002 is greater than the number of indicators that showed a decrease in
compliance between the 2 years.
Question 4: For each indicator category, is there a significant difference in the magnitude of
change (across all indicators in the category)in the proportion of compliant plants
between 2001 and 2002?
Statement of Null and Alternative Hypotheses:
Ho: The magnitude of change in the proportion of compliant plants between 2001 and 2002
across all indicators in an indicator category is not statistically different from zero.
HA: The magnitude of change in the proportion of compliant plants between 2001 and 2002
across all indicators in an indicator category is statistically greater than zero.
The analysis for Questions 3 and 4 required that the data be arranged as shown in the following table.
y Proportion of Plants in Sign(Direction)of
Indicator Compliance Change Change
Indicator
Category 2001 2002 (2002-2001) ("+", ,0)
„ ,
air,hazardous
waste,etc. Number 0.40 0.60 0.20 +
Question 3 was addressed using the sign test,which is a non-parametric test that compares the number of
indicators showing an increase in compliance between years(noted using the"+" symbol in the example
table above)with the number of indicators that show a decrease(noted using the"" symbol). Question 3
uses a one-sided hypothesis test,which specifically addresses the question of whether there was a
statistically significant improvement in compliance between years The significance of the test is based on
the binomial distribution,which is used to calculate the number of"+" signs that would be expected by
Final Report 2-11 Asphalt Plant Compass Project
chance alone if there was no increase in compliance between years. The results of this test are provided
in Section 2.4 and Appendix A-3,Table A-3-2.
The approach for Question 4 is similar to Question 3 in that it also uses the"paired"observations between
years;however,the focus for Question 4 is on the relative magnitude of change in the proportion of
compliant plants between years. For this analysis,the change in the proportion of plants showing
compliance(that is,proportion compliant in 2002 minus proportion compliant in 2001) is calculated for
each indicator within each indicator category. Parametric(paired t-test)and non-parametric(Wilcoxon
sign-rank test)tests were used to determine whether the overall change was significantly different from
zero. The results of these tests are provided in Section 2.4 and Appendix A-3,Table A-3-3.
Analyzing the asphalt manufacturing inspector checklist data required eliminating indicators with a NR
(no response)or NA(not applicable)response. While this action was necessary to"clean up"the data in
order to perform the statistical analysis,doing so can introduce bias into the results. For example,
throwing out NR responses can skew the data because the actual responses may have been `Yes' (in
compliance),No(not in compliance),or NA. A bias is also introduced as a result of removing the NA
responses;however,the bias is difficult to predict.
2.4 DATA ANALYSIS RESULTS
APCD analyzed data both quantitatively using statistics and descriptively using charts and graphs to
represent the results. Both the quantitative and descriptive results include analysis of the data by indicator
and by plant. Some of the analyses considered the data grouped into indicator categories(air quality,
storm water,hazardous waste,and pollution prevention)while others evaluated the data on an individual
indicator level. This section summarizes the quantitative and descriptive data analysis results by indicator
and by plant.
The descriptive results are presented graphically in a series of pie and bar charts located throughout this
section. There are seven charts for each indicator category as follows:
Analysis by Indicator
I. A pie chart depicting the distribution of COI's, (described in Section 2.2,Task 6)within an
indicator category divided into four result groupings(increase in compliance,decrease in
compliance,no change with high initial compliance,and no change with low initial
compliance). See the description of result groups below. (Figures 2-4, 2-11, 2-18,and 2-25)
2. A bar chart showing the percent change in frequency of compliance from baseline for each
CCI in the indicator category. (Figures 2-5, 2-12,2-19, and 2-26)
r
Final Report 2-12 Asphalt Plant Compass Project
3. A pie chart depicting the distribution of non-critical indicators within an indicator category
divided into four possible result groups(increase in compliance,decrease in compliance,no
change with high initial compliance,and no change with low initial compliance). (Figures
2-6, 2-13,2-20,and 2-27)
4. A bar chart showing the percent change in frequency of compliance from baseline for each
non-critical indicator in the aggregate group. (Figures 2-7,2-14a,2-14b, 2-21,2-28a,and 2-
28b)
Analysis by Plant
1. A pie chart depicting the distribution of plants within an indicator category divided into four
possible result groups(increase in compliance,decrease in compliance,no change with high
initial compliance,and no change with low initial compliance). (Figures 2-8,2-15, 2-22,and
2-29)
2. A bar chart showing the percent change in frequency of compliance from baseline for each
plant in the indicator category. (Figures 2-9a,2-9b,2-16, 2-23,and 2-30)
3. A bar chart showing the distribution of compliance rates of plants for the indicator category.
(Figures 2-10,2-17, 2-24,and 2-31)
Result Groups Description
Increase Increase in compliance,with a subgroup of indicators with a
statistically significant increase
Decrease Decrease in compliance,with a subgroup of indicators with a
statistically significant decrease
No change—high at baseline No change in compliance,with a high initial compliance*
No change—low at baseline No change in compliance,with a low initial compliance*
*For the"no change" groups,high initial compliance was defined as greater than 50 percent of
the plants in compliance for a particular indicator. Low initial compliance was defined as
less than 50 percent of the plants in compliance for a particular indicator.
Analysis by Indicator Category
APCD analyzed the data by indicator category with two questions,Question 3 and Question 4. These
analyses determined whether there was a significant difference in(1)the number of indicators across an
indicator category that increased from baseline to follow-up and(2)the magnitude of change across the
indicator category from baseline to follow-up. APCD ran these tests for five indicator categories: air
quality, storm water,hazardous waste,pollution prevention,and all indicators. Both statistical tests show
a statistically significant difference for all indicator categories. Table A-3-3 in Appendix A-3 contain the
statistical analysis results. Table 2-6 in Section 2.5 summarizes the results.
2.4.1 Air Quality Results
The statistical analysis by indicator(Question 1)for air quality showed that nine indicators increased in
performance and no indicators decreased in performance in a statistically significant manner. Three of
the statistically significant increases were CCIs, listed below:
Final Report 2-13 Asphalt Plant Compass Project
• Indicator 71 (I71): "Is the opacity from the hot asphalt storage bin below 20 percent?"
• I148: "Storage Piles: Is the fugitive control plan submitted being followed?"
• 1161: "Roads: Is the control plan adequate?"
The results for each indicator are included in Appendix A-3,Table A-3-1. Figures 2-4 through 2-7,
located at the end of Section 2.0, graphically represent the quantitative analysis by indicator. Appendix
A-4 contains a key to match indicator numbers with the questions they represent.
The statistical analysis by plant(Question 2)for air quality showed that 10 asphalt plants exhibited a
statistically significant increase in performance and one plant exhibited a statistically significant decrease
in performance among air quality indicators. The results for each plant are included in Appendix A-3,
Table A-3-1. Figures 2-8 through 2-10, located at the end of Section 2.0, graphically represent the
quantitative analysis by plant.
2.4.2 Storm Water Results
The statistical analysis by indicator(Question 1)for storm water showed that 21 indicators increased in
performance and no indicators decreased in performance in a statistically significant manner. Six of the
statistically significant increases were CCI's,listed below:
• I227: "Storm Water Management Plan available on site?"
• I258: "Preventive maintenance program adequate?"
• 1261: "Good housekeeping program adequate?"
• 1267: "Spill program adequate?"
• 1269: "Training program adequate?"
• 1273: "Assessment program adequate?"
The results for each indicator are included in Appendix A-3,Table A-3-1. FigureQ- 11 through 2-14,
located at the end of Section 2.0, graphically represent the quantitative analysis by indicator. Appendix
A4 contains a key to match indicator numbers with the questions they represent. Refer to Section 2.4 for
a general explanation of the attached charts and Figures 2-4 through 2-7 for an example of specific
interpretation.
The statistical analysis by plant(Question 2)for storm water showed that 13 asphalt plants exhibited a
statistically significant increase in performance and none a statistically significant decrease in
performance among storm water indicators. The results for each plant are included in Appendix A-3,
Table A-3-1. Figures 2-15 through 2-17, located at the end of Section 2.0, graphically represent the
quantitative analysis by plant. Refer to Section 2.4 for a general explanation of the attached charts and
Figures 2-8 through 2-10 for an example of specific interpretation.
Final Report 2-14 Asphalt Plant Compass Project
2.43 Hazardous Waste Results
The statistical analysis by indicator(Question 1)for hazardous waste showed that one indicator(1465: "Is
training provided for Universal Waste?")increased in performance and no indicators decreased in
performance in a statistically significant manner. The one statistically significant increase was not a CCI.
The results for each indicator are included in Appendix A-3,Table A-3-1. Figures 2-18 through 2-21,
located at the end of Section 2.0, graphically represent the quantitative analysis by indicator. Appendix 4
contains a key to match indicator numbers with the questions they represent. Refer to Section 2.4 for a
general explanation of the attached charts and Figures 2-4 through 2-7 for an example of specific
interpretation.
The statistical analysis by plant(Question 2)for hazardous waste showed that no asphalt plants exhibited
a statistically significant increase or decrease in performance among hazardous waste indicators. The
results for each plant are included in Appendix A-3,Table A-3-1. Figures 2-22 through 2-24,located at
the end of Section 2.0, graphically represent the quantitative analysis by plant. Refer to Section 2.4 for a
general explanation of the attached charts and Figures 2-8 through 2-10 for an example of specific
interpretation.
2.4.4 Pollution Prevention Results
The statistical analysis by indicator(Question 1)for pollution prevention showed that four indicators
increased in performance and four indicators decreased in performance in a statistically significant
manner.
Statistically Sivtificant Increase Statistically Sienificant Decrease
• 1297: "Do you share tracked fuel use • 1279: "Did you use the laboratory
information with co-workers and the burner mixing temperature as the plant mixing
manufacturer?" temperature?"
• 1312: "Do you contact the asphalt supplier, • 1303: "Do you measure and record the
describe the mix type,and request the plant pressure drop in the baghouse and look
mixing temperature recommendations?" for changes over time?"
• 1322:"Do you use stockpiling techniques • 1327:"Do you use RAP that contains
(place material in a small are and stacking as tar?"
high as possible without risking • 1383:"Do you try to pave when ambient
contamination)to allow materials to shed weather conditions are conducive to low
rain?" emissions(high wind velocity,low
• 1366"Is your output at 100%burner close to relative humidity,low dew point)?"
faceplate rating?"
Final Report 2-15 Asphalt Plant Compass Project
None of the statistically significant increases or decreases were CCI's. The results for each indicator are
included in Appendix A-3,Table A-3-1. Figures 2-25 through 2-28b,located at the end of Section 2.0,
graphically represent the quantitative analysis by indicator. Appendix A4 contains a key to match
indicator numbers with the questions they represent. Refer to Section 2.4 for a general explanation of the
attached charts and Figures 2-4 through 2-7 for an example of specific interpretation.
The statistical analysis by plant(Question 2)for pollution prevention showed that three asphalt plants
exhibited a statistically significant increase in performance and three other asphalt plants exhibited a
statistically significant decrease in performance among pollution prevention indicators. The results for
each plant are included in Appendix A-3,Table A-3-1. Figures 2-29 through 2-30,located at the end of
Section 2.0,graphically represent the quantitative analysis by plant. Refer to Section 2.4 for a general
explanation of the attached charts and Figures 2-8 through 2-10 for an example of specific interpretation.
2.5 SUMMARY OF FINDINGS
This section summarizes the results of statistical tests described in Section 2.3 and presented in 2.4 to
compare(1)responses to individual indicators between 2001 and 2002(Questions 1 and 2)and(2)
differences in indicator categories between 2001 and 2002(Questions 3 and 4).
Results for individual indicators(Question 1)are organized according to the following four groups:
1. Increase in compliance,with a subgroup of indicators with a statistically significant increase
2. Decrease in compliance,with a subgroup of indicators with a statistically significant decrease
3. No change in compliance,with a high initial compliance
4. No change in compliance,with a low initial compliance
For the"no change" groups,high initial compliance was defined as greater than 50 percent of the plants
in compliance for a particular indicator. Low initial compliance was defined as less than 50 percent of the
plants in compliance for a particular indicator. The individual indicator comparison results for air quality,
hazardous waste,storm water,and pollution prevention are summarized in Tables 2-1 through 2-4,
respectively. Appendix A-4 contains a key to match indicator numbers with the questions they represent.
Final Report 2-16 Asphalt Plant Compass Project
TABLE 2-1
STATISTICAL ANALYSIS SUMMARY
FOR INDIVIDUAL AIR QUALITY COMPLIANCE INDICATORS
Analysis Groups and Indicator
Number Of Indicators (CCI in Brackets [])
Increase in Compliance 22 [12], [18], [42], [58], [63], [66], [69], 72,77, 144, 143,
145, 147, 154, 155, 156, 157, 159, [160], [659], 1665]
Statistically Significant Increase 9 6, 7, [71],73,74, [148], 151, 153, [161]
Decrease in Compliance 6 [40],65,41,65,41,64, [142], 146
Statistically Significant Decrease 0 None
No Change 0 None
High Initial Compliance
No Change 0 None
Low Initial Compliance
TABLE 2-2
STATISTICAL ANALYSIS SUMMARY
FOR INDIVIDUAL STORM WATER COMPLIANCE INDICATORS
Analysis Groups and Indicator
Number Of Indicators (CCI in Brackets[])
Increase in Compliance 6 228,233, 234,236,237, 239
Statistically Significant Increase 21 232, 225, [227], 256,257, [258],259,260, [261], 262,
263, 264,265,266, [267],268, [269], 270,271,272,
[273]
Decrease in Compliance 7 231, 235,238,243,247,251, 253
Statistically Significant Decrease 0 None
No Change I 252
High Initial Compliance
No Change 0 None
Low Initial Compliance
TABLE 2-3
STATISTICAL ANALYSIS SUMMARY
FOR INDIVIDUAL HAZARDOUS WASTE COMPLIANCE INDICATORS
Analysis Groups and Indicator
Number Of Indicators (CCI in Brackets [])
Increase in Compliance 6 456,457, [459],461,463,461
Statistically Significant Increase 1 465
Decrease in Compliance 2 [458],468
Statistically Significant Decrease 0 None
No Change 0 None
High Initial Compliance
No Change 0 None
Low Initial Compliance
Final Report 2-17 Asphalt Plant Compass Project
TABLE 2-4
STATISTICAL ANALYSIS SUMMARY
FOR INDIVIDUAL POLLUTION PREVENTION INDICATORS
Analysis Groups and Indicator
Number Of Indicators (CCI in Brackets [])
Increase in Compliance 41 278, 279, [287],289, 294, [295], 297,298,300,
301,312, [315],317, 318,319,320, 322,324,
[325],326,327,330,331,332,333,335,337,
338,340, [348],351,353,355,359,363,365,
366,376,378,379,384,
Statistically Significant Increase 4 297, 312, 322,366
Decrease in Compliance 13 [285],305,313,314,316,321,328,329,336,
367,380,381,382
Statistically Significant Decrease 4 279,303,327,383
No Change 3 339,357, 361
High Initial Compliance
No Change 0 None
Low Initial Compliance
Table 2-5 summarizes the statistical analysis results for each plant within an indicator group(Question 2).
TABLE 2-5
INDICATOR CATEGORY ANALYSIS RESULTS-(QUESTION 2)
Number of Plants with Number of Plants
Statistically Significant with Statistically
Indicator Category Increase Significant Decrease
All indicators 22 0
Air quality indicators 10 1
Storm water indicators 13 0
Hazardous waste indicators 0 0
Pollution prevention indicators 3 3
Table 2-6 summarizes the statistical analysis results for each indicator category(Questions 3 and 4),all of
which were statistically meaningful.
TABLE 2-6
INDICATOR CATEGORY ANALYSIS RESULTS-(QUESTIONS 3 and 4)
Baseline Venus Follow Up
✓= statistically significant increase
Indicator Category z=no significant change
Number of Indicator Magnitude of Change
(Question 3) (Question 4)
All indicators ✓ ✓
Air quality ✓ ✓
Storm water ✓ ✓
Hazardous waste ✓ ✓
Pollution prevention ✓ ✓
r
Final Report 2-18 Asphalt Plant Compass Project
The statistical analysis of baseline versus follow-up results suggest the following overall conclusions
about the asphalt plant Compass project between 2001 and 2002:
✓ Overall compliance for asphalt plants significantly improved. 109 indicators(75 percent of
all indicators)increased in performance.
✓ Thirty four indicators increased in a statistically significant manner.
✓ Thirty indicators(25 percent of all indicators)decreased in performance.
✓ Four indicators decreased in a statistically significant manner;none were compliance related,
all were pollution prevention indicators.
✓ Twenty-nine critical compliance indicators(CCI)were identified;9 CCI's showed
statistically significant improvement and no CCIs decreased in performance.
While the statistics offer compelling evidence supporting the asphalt plant Compass project's positive
impact on compliance,descriptive information from the assessments provide additional insight into each
sector's environmental performance(with respect to regulatory compliance). Table 2-7 lists average
compliance"scores"for each aggregate group for each indicator category. A compliance score is the
percent of indicators for which an"in compliance"answer was noted on the checklist on a 10 to 100
percent scale. For example,a compliance score of 75 percent indicates that a plant was in compliance
with 75 percent of all indicators. An average indicator category compliance score of 75 percent indicates
that,on average,all facilities were in compliance with 75 percent of the indicators within that indicator
category.
TABLE 2-7
AVERAGE COMPLIANCE SCORES
Average Compliance Score
Indicator Category Basrline Follow-up Increase
Air quality 73% 88% 15%
Storm water 48% 79% 31%
Hazardous waste 47% 54% 7%
Pollution prevention 76% 78% 2%
Note: Change in average scores was not statistically analyzed;therefore,no statistical significance
should be assigned to these changes.
Final Report 2-19 Asphalt Plant Compass Project
Figure 2-4
Baseline to Follow-up Compliance Change Aggregate Mr Quality
Analysis for Critical Compliance Indicators
13% Among the air quality
10 CCI,87 percent of the
indicators increased in
compliance and 13
percent decreased
•Increase in Compliance Rate from 2001 to 2002.
•Decrease in Compliance Rate
87%
Figure 2-5
Baseline to Follow-up Compliance Change
Indttidual Indicator Air Quality Analysis
for Critical Compliance Indicators
1
too — -- - i This chart depicts which
so j specific air quality CCI
2 60 I increased or decreased
m40
i " from 2001 to 2002 and
�' 20 — how much they changed.
c., 0 `- Yellow bars indicate that
_
20 R 18 40 42 58 63 66 69 71 148 160 151 659 665 a change was statistically
w significant(indicators 71,
-40 148, 160,and 161).
.60 _ _..._ I
Indicator
IllNot a statistically significant change
El Statistically significant change
Final Report 2-20 Asphalt Plant Compass Project
Figure 2-6
Baseline to Follow-up Compliance Change Aggregate
Air Quality Analysis fo r Non-Critical Complaince
indicators
Among the air quality
19% non-critical
111 indicators,81 percent
• Increase in Compliance of the indicators
increased in
•Decrease in Compliance compliance
eant decre
d 19
ased
sed
percent
from 2001 to 2002.
81%
Figure 2-7
Baseline to Follow-up Compliance Change
Indisidual Indicator Air Quality Analysis ty Al y'sis This chart depicts which
for Critical Compliance Indicators specific air quality non-
100 critical compliance
80 indicators increased or
0 60 decreased from 2001 to
�— 2002 and how much
40
c .5 — they changed. Yellow
20 bars indicate that a
c._) m
a 0 change was statistically
P. -20 U 18 40 42 58 63 66 69 71 148 160 161 659 665 significant(indicators 6,
7,73,74, 151,and 153).
-40
-60
Indicator
Not a statistically significant change
Statistically significant change
El
Final Report 2-21 Asphalt Plant Compass Project
Figure 2-8 1
Baseline to Follow-up Compliance Change 78 percent of the plants increased
Aggregate Mr Quality Analysis compliance and 10 percent
12% for Plants decreased compliance among all
air quality indicators from 2001 to
2002. Twelve percent of the
l0% •Increase in Compliance plants had the same percent
compliance with air quality
•Decrease in Compliance indicators in 2001 and 2002 and
•No Change in Compliance the compliance was higher than 50
(high at baseline) percent in 2001 (baseline). The
increase in compliance among air
Overall increase in compliance is quality indicators was significant
statistically significant(number of in terms of the number of
78% indicators and magnitude of change) indicators and magnitude of
change.
Slle
Figure 2-9a
Baseline to Follow-up Compliance Change
Individual Plant Air Quality Analysis
100
80
0 60
qw .5 40
o
o 20
o 0 0 0
a4 es. -20 Figures 2-9a and 2-9b
U -40 depict which specific
-60 _..--- — — plants increased or
Plant decreased in compliance
from 2001 to 2002 and
how much they changed.
Figure 2-9b Yellow bars indicate that
Baseline to Follow-up Compliance Change the change was
Individual Plant Air Quality Analysis statistically significant
100 (plants 4,5,9,20,22,
8 80 25,34, 36,38,41,and
o N 60 42).
>; w .E 40 _ imiiii
°' 20 w to g PPalli-11
V -20 r•wa,to— oiv a�wOav�tII'i .Da�w 0lv7+t4
40 .-- --ANN NNNNNNNrn'n ntntntntn ntn n4�4(
-60
Plant
• Not a statistically significant change
❑ Statistically significant change
A.—^ Final Report 2-22 Asphalt Plant Compass Project
Figure 2-10
Baseline to Follow-up Distribution o f
Aggregate Air Quality
Co mpliance Among All Plants
,n 100 ■Baselne Average =73%
80 D Follow-up Average=88%
w, 60
° 40
0 20 —_^
a 0 I
0 10 20 30 40 50 60 70 80 90 100
Aggregate Compliance (percent)
Chart represents snapshot of plant
compliance change from 2001 to 2002.
Green bars represent plant compliance
in 2001 and yellow bars represent plant
compliance in 2002. For example,in
2001 (baseline) 19 percent of the plants
were in compliance with 60 to 69
percent of all air quality compliance
indicators. The baseline and follow-up
averages refer to average compliance of
all plants for the year. For example,in
2002(follow-up),on average,plants
were in compliance with 88 percent of
the air quality indicators.
Final Report 2-23 Asphalt Plant Compass Project
Figure 2-11
Baseline to Follow-up Compliance Change
Aggregate Storm Water Analysis
for Critical Compliance Indicators
•Increase in Compliance
Oloose
Figure 2-12
Baseline to Follow-up Compliance Change
Individual Indicator Storm Water Analysis
for Critical Compliance Indicators
100
as c 80
lc °U 40
a ill 0
20
-20 277 258 261 967 769 971 I
o a4 -60
Indicator
O Statistically significant change
Figure 2-13
Baseline to Follow-up Compliance Change
Aggregate Storm Water Analysis for Non-Critical
Compliance Indicators
3%
i%24 • Increase in Compliance
•Decrease in Compliance
•No Change in Compliance
(high at baseline)
73%
'^ Final Report 2-24 Asphalt Plant Compass Project
Figure 2-14a
Baseline to Follow-up Compliance Change Individual
Indicator Storm Water Analysis for Non-Critical
Compliance Indicators
100 -
80
aE 60
c w a 40
a 20
-20 N en n r ee
V
-4o N N N en en en en en en en en V I
av N N (NI av aV aV aV N al a N N
-60 -
Indicator
Figure 2-14b
Baseline to Follow-up Compliance Change Individual
Indicator Storm Water Analysis for Non-Critical
Compliance Indicators
100 -
80
O N 60
a `+ 5 40 —
°' 20
oo ti
I 0
0.1
U -20 N M r Ol O CJ M .Y vl ,$) '.0 O . N
d' teHC1 leiN tinVl 1/44 aD N N � N Cl N N
-40 N ClCJ ClN CJ CJ ClN aJ- aJ N N N Cl N Cl N N aV
-60 -
—
Indicator
NI Not a statistically significant change
Statistically significant change
Final Report 2-25 Asphalt Plant Compass Project
Figure 2-15
Baseline to Follow-up Compliance Change Aggregate
Storm Water Analysis for All Plants
12% •Increase in Compliance
•Decrease in Compliance
18% •No Change in Compliance
(high at baseline)
❑No Change in Compliance
61% (low at baseline)
9%eOverall increase in performance is
statistically significant(number of
questions and magnitude of change)
Figure 2-16
Baseline to Follow-up Compliance Change
Individual Plant Storm Water Analysis
100
§ 80
60
ll
1 40
1 S 20
Utzl
0
8 -20 3 5 8 11 13 15 18 21 23 25 27 29 31 34 37 41
d° -40
-60
Plant
II Not a statistically significant change
Ei Statistically significant change
Figure 2-17
Baseline to Follow-up Distribution of
Aggregate Stone Water Compliance
Among All Plants •Baseline
100 - Average=48%
i S0 O Follow-up
a. 60 . Average=79%
c.
o I
1g 40 I
aB. 20
0 � ,. , ~ , • , � , • • , � � ,
0 10 20 30 40 50 60 70 80 90 100
Aggregate Compliance(percent)
Final Report 2-26 Asphalt Plant Compass Project
Figure 2-19
Figure 2-18 Baseline to Follow-up Compliance Change
Baseline to Follow-up Compliance Change Individual Indicator Hazardous Waste Analysis for
Aggregate Hazardous Waste Analysis Critical Compliance Indicators
for Critical Complaince Indicators
100
80
° 60
m C
s 40
■Increase in Compliance U ffi 20
50% 50% E 0
°
•Decrease o Compliance w 20
-40
-60
Indicator
Not a statistically significant change
Figure 2-20 Figure 2-21
Baseline to Follow-up Compliance Change Baseline to Follow-up Compliance Change
Aggregate Hazardous Waste Analysis Individual Indicator Hazardous Waste Analysis for Non-
^ for Non-Critical Compliance Indicators Critical Compliance Indicators
14% 8 100
80
600 40■Increase in Compliances.
3 ---
b u 20
0
■Decrease wCompliance -20
p -40
huicator
II Not a statistically significant change
86% 0 Statistically significant change
Final Report 2-27 Asphalt Plant Compass Project
Figure 2-22
Baseline to Follow-up Compliance Change
Aggregate Hazardous Waste Analysis
for All Plants
8%
■Increase in Compliance
ax
■Decrease in Compliance
43%
•No Change in Compliance
(high at baseline)
❑No Change in Compliance
(low at baseline)
3 t%
Figure 2-23
Baseline to Follow-up Compliance Change
Individual Plant Hazardous Waste Analysis
too
• 80
60
0 40
O 20
F 0
-20 sr o en m v
-40
a.
Plant
■ Not a statistically significant change
Figure 2-24
Baseline to Follow-up Distribution of Aggregate
Hazardous Waste Among MI Plants
•Baseline
100 Average=47%
a,
" 80 ❑Follow-up
E 60 Average=54%
40 P. 0
20 1. a.-- MI MI -
0 10 20 30 40 50 60 70 80 90 100
Aggregate Compliance (percent)
Final Report 2-28 Asphalt Plant Compass Project
Figure 2-25
Baseline to Follow-up Compliance Change
Aggregate Pollution Prevention Analysis
17% for Critical Complaince Indicators
Ili •Increase in Compliance
•Decrease in Compliance
83%
Figure 2-26
Baseline to Follow-up Compliance Rate Change
Individual Indicator Pollution Prevention Analysis
for Critical Compliance Indicators
100
0fi 80
w 60
g 9 40
,y g 20
U el0
P -20
a -40
-60
Indicator
INot a statistically significant change
Figure 2-27
Baseline to Follow-up Compliance Change Aggregate
Pollution Prevention Analysis for Non-Critical
Complaince Indicators
6%
•Increase in Compliance
•Decrease in Compliance
25%
•No Change in Compliance(high
69% at baseline)
^ Final Report 2-29 Asphalt Plant Compass Project
Figure 2-28a
Baseline to Follovt-up Compliance Change
Individual Indicator Pollution Prevention Analysisfor
Non-Critical Compliance Indicators (first half)
100 --
80
60
uta O 40
dA 20 ��iTr1 1■��
00 O, O. 7 1- M V1 N M d' (� 00 O. O, N V ‘0 1, •• rn 0
0. -20 N 1"- N N N N M M u m M M M M m m e m m m m m m m M M
-40
-60
Indicator
Not a statistically significant change
❑ Statistically significant change
Figure 2-28b
Baseline to Follow-up Compliance Rate
ChangeIndividual Indicator Pollution Prevention
Analysisfor Non-Critical Compliance Indicators (second
halt)
100
80
60
0 40
ea 'r —
20
U
m 0 PI Ili
-.Nm b N 00O.O,,,,,mv N —enw .O 00O.O-Nm7
-20 M M M M M M m 7 V 1A,%;, n�� � 'ON N N 00 00 00 00 00
m m m m M M M M M M M M M M M M M M M M M M M M m m
-40
-60
Indicator
Not a statistically significant change
❑ Statistically significant change
Final R.,purt 2-30 Asphalt Plant Compass Project
Figure 2-29
Baseline to Follow-up Compliance Change
Aggregate Pollution Prevention Analysis
for All Plants
13% ■ Increase in Compliance
■ Decrease in Compliance
IN No Change in Compliance
30% 57% (high at baseline)
Overall increase in performance is
statistically significant (number of
questoins and magnitude of change)
Figure 2-30
Baseline to Follow-up Compliance Change
Individual Plant Pollution Prevention Analysis
100 -
80
a 60
o 40
u
um 20
Q W 0 m Illrn �
�i -c rnv�r o r000�a�cmavno cvdv eor000 re r�
-20c x�ca mKrnmarxnv�ty
-40
-60 -
Plant
Final Report 2-31 Asphalt Plant Compass Project
Figure 2-31
Baseline to Follow-up Distribution of Aggregate
Pollution Prevention Among All Plants
100 — • Baseline Average =76%
80 — Follow-up Average= 78%
F., 60 —
o
a
40
� 20
0 10 20 30 40 50 60 70 80 90 100
Aggregate Compliance (percent)
Final Report 2-32 Asphalt Plant Compass Project
3.0 HAZARDOUS WASTE SMALL QUANTITY GENERATORS
RESULTS AT A GLANCE
To test the effectiveness of delivering compliance assistance before an inspection,the Colorado
Department of Public Health and Environment(CDPHE),Hazardous Materials and Waste
Management Division(HMWMD),selected 49 hazardous waste small quantity generators(SQG)in
Colorado and divided them into three groups. Each group received a different level of notification and
compliance assistance before HMWMD conducted a hazardous waste inspection. After the
inspections,HMWMD noted hazardous waste violations in 39 categories of violations(subsequently
referred to as coverage areas). HMWMD conducted two statistical tests on the data collected during
the inspections: 1)an evaluation of the overall compliance rates per facility between groups 1,2,and
3;and 2)an evaluation of the rates of compliance for selected coverage areas and for all facilities and
all 39 coverage areas. The results of the first test show that the average rate of compliance per facility
is higher for both groups 2 and 3 compared to group 1,but none of the differences between groups are
statistically significant.
In the second test,nine coverage areas were evaluated. HMWMD found statistically-significant
differences in compliance rates between groups for three coverage areas. In each instance,rates of
compliance were higher in groups 2 and 3 relative to group 1. However,in all three cases,the results
are suspect because the sample sizes were too low,which biased the statistical comparisons.
HMWMD also evaluated compliance rates taking into account the responses of all facilities for all 39
coverage areas combined. The results of this analysis indicate a highly significant difference among
groups. Overall compliance rates in group 1 were statistically lower compared to both groups 2 and 3,
but compliance rates were not significantly different between group 2 and group 3.
Based on these analyses,HMWMD concludes that the Compass project showed that inspection
notification has a net positive effect on SQG compliance rates. While small sample-sizes and other
limitations of the survey design did not allow for an optimal test of the effectiveness of notification at
the level of an individual facility or coverage area,when the results for all facilities and coverage areas
are evaluated together,the overall effect of notification is clearly shown. The two levels of
notification were shown to be approximately equivalent in their effectiveness, suggesting that there is
no added benefit to implementing the higher level of pre-inspection notification.
HMWMD believes that better results could have been attained at the individual coverage area level
with:a larger sample size;a better survey design;better pm-screening of facilities for SQG status;and
more complete collection of inspection data.
Based on the descriptive data inspectors collected,HMWMD noted a possible relationship between
the presence of an environmental coordinator at a facility and higher compliance and the presence of
an EMS at a facility and higher compliance. Conversely,there seems to be no relationship between
compliance rates and the presence of a compliance assistance program at a facility.
HMWMD gathered data on the resources were spent in creating and delivering the intervention
activities. However,because statistical results indicated no statistically significant difference in
compliance between groups 2 and 3,HMWMD did not evaluate the cost-benefit of the two levels of
intervention.
Final Report 3-1 SQG Compass Project
3.1 PROJECT DESCRIPTION
The main purpose of the HMWMD's SQG Compass project was to test the effectiveness of notifying
SQGs of an upcoming inspection and delivering compliance assistance to facilities before an inspection.
Specifically,HMWMD wanted to test the effectiveness of two activities for increasing compliance: (1)
simply notifying a hazardous waste generator of an upcoming inspection,and(2)providing the SQG with
the same notification of an upcoming inspection along with extensive written compliance assistance and
guidance. To test these activities,HMWMD chose a universe of regulated facilities that it had not
recently inspected(within the last 2 years). HMWMD wanted a universe with a low inspection history to
ensure that previous inspections had not already had a positive effect on compliance rates. Because
Colorado SQOs include industry sectors with a history of few or no inspections,several SQGs industry
sectors were chosen for the pilot project.
HMWMD initially chose 60 facilities from three sectors of SQGs for the HMWMD's SQG Compass
project. HMWMD planned to place the 60 facilities into one of three groups with each group receiving
different levels of inspection notification and compliance and pollution prevention assistance (as
explained further in Section 3.2). Choosing a sample size of 60 facilities was somewhat arbitrary; it
allowed HMWMD to place an equal number of facilities(20)in each of the three groups,which
HMWMD thought would be an adequate sample size,but that did not overwhelm available resources
(that is, inspection time). HMWMD inspected each facility between late 2001 and mid 2002.
3.2 PROJECT IMPLEMENTATION APPROACH
The SQG Compass project involved five main tasks as follows:
• Enhance Notice of Inspection form
• Select SQG industry sectors for inclusion in HMWMD's SQG Compass project
• Prepare compliance assistance packets for each group 3 SQG
• Contact and Inspect Compass SQGs
• Collect and analyze data
Each of these tasks are discussed in further detail in the remainder of Section 3.2.
Task 1: Enhance Notice of Inspection Form
HMWMD records hazardous waste inspection violation information on notice of inspection(NOI)forms
that HMWMD inspectors complete in the field at the close of each inspection. Before Compass,
HMWMD inspectors recorded all hazardous waste violations on the NOIs in one of seven comprehensive
violation categories(called`coverage areas'). In this task,HMWMD divided the seven original coverage
areas into 39 more detailed coverage areas,each associated with one regulatory requirement or a small
group of closely related requirements. HMWMD created the 39 coverage areas based on its evaluation of
Final Report 3-2 SQG Compass Project
past compliance data,which indicated that the seven original coverage areas were not detailed enough to
provide information on the specific regulations hazardous waste generators were violating. For instance,
the original coverage areas grouped all container management violations together. The new coverage
areas include separate coverage areas for each container requirement(closed containers,labeling
requirements,aisle space,etc.). By differentiating the coverage areas,HMWMD hoped to more
accurately identify regulatory violations most common among SQGs,better understand SQG compliance
problems,and design and deliver better-targeted compliance assistance.
Task 2: Select SQG Industry Sectors for Inclusion in SQG Compass Project
HMWMD began selecting the SQG sectors for the Compass project in early 2001. HMWMD hoped to
select about 60 SQGs for the Compass project because it thought that sample size would be large enough
to detect statistical significance`and small enough to implement the project with the resources available.
HMWMD needed SQG sectors that bad little to no previous inspection history because previous
compliance input may have skewed the baseline data. As explained in Section 3.1,HMWMD felt that
facilities that had recently been inspected would have better compliance if reinspected than a facility that
had not been recently inspected. While HMWMD was in the process of determining the SQG sectors to
include in the project,it met with U.S. Environmental Protection Agency(EPA)Headquarters(HQ)staff
to discuss some technical design aspects of the SQG Compass project. In that meeting,EPA indicated
that,based on its experience in attempting to accurately estimate the compliance rate of a sector,
HMWMD should target an entire sector that could be inspected during the project. Before the
conversation with EPA,HMWMD planned to use SQG facilities that had never before been inspected,
but not confine its project participants to one sector or entire sectors. Based on the EPA HQ feedback,
HMWMD changed its design to focus on one or several sectors that HMWMD could inspect in its
entirety and that included up to 60 SQGs.
Several unexpected bathers developed when HMWMD identified sectors for the SQG Compass project.
The first barrier was that no Standard Industrial Classification(SIC)or North American Industrial
Classification System(NAICS)codes had been assigned to these facilities in HMWMD's database.
Therefore,HMWMD began assigning SIC codes to SQGs based on company name and any other
information available such as file information or information included on the facility's hazardous waste
notification form. After assigning SIC codes, HMWMD sorted the SQG universe into industry sectors
and evaluated likely sectors to fmd those with about 60 members and a history of low inspection rates.
HMWMD found no single sector that met both requirements. Therefore,HMWMD looked for a group of
c HMWMD did not consult with a survey statistician to determine the number of SQGs to survey. Thus,HMWMD
did not have a scientific basis for its decision to select 60 SQGs.
Final Report 3-3 SQG Compass Project
sectors with low inspection histories that each included less than 60 SQGs,but added up to about 60
SQGs. On the basis of this evaluation,HMWMD chose the following sectors of SQGs for the Compass
project.
• Chemical and allied products
• Transportation equipment
• Air transportation
• Wood products
• Furniture and fixtures
HMWMD randomly placed the SQGs from these five sectors into one of three groups as explained in
Table 3-1.
TABLE 3-1
SQG GROUP ASSIGNMENTS
Group Description
1 "Control group"; HMWMD notified SQGs of inspections in the historically
typical manner;that is,no prior contact with the facility other than the call
scheduling the inspection 5 to 10 working days in advance.
2 HMWMD sent a letter in advance of the inspection notifying the SQGs that
HMWMD would conduct an inspection to evaluate compliance with Colorado
Hazardous Waste Regulations at the facility sometime within 6 months.
3 HMWMD sent the same letter to group 3 as it sent to group 2. In addition,
HMWMD sent all compliance guidance documents it had that were relevant to
the SQG operations and a pollution prevention fact sheet relevant to the SQGs
operations.
The only exception to the random group placement was that HMWMD automatically placed any SQG
that had been inspected within the previous 2 years in group 1 (control group). HMWMD felt that
inspection data from inspections that occurred within the past 2 years was recent enough to be relevant to
the Compass project. Further,HMWMD though that if it re-inspected SQGs it had inspected in the past 2
years,the compliance would be higher than during the previous inspection. The higher compliance would
inaccurately inflate the compliance of the facility and therefore the results of whichever group the facility
had been assigned. HMWMD used the results of the existing inspection rather than conducting an
inspection during the SQG Compass project. Because the existing inspection data categorized violations
into the original seven coverage areas,HMWMD looked at the inspection forms and reassigned violations
to the 39 coverage areas.
Task 3: Prepare Compliance Assistance Packets for Each Group 3 SQG
HMWMD created four pollution prevention fact sheets for the group 3 SQGs. The topics of the fact
sheets included chemical and allied products,wood products and furniture and fixtures,aqueous cleaning,
and surface coating.
Final Report 3-4 SQG Compass Project
HMWMD then prepared a customized information packet for group 3 SQGs that included the pollution
prevention fact sheets and any hazardous waste guidance documents applicable to manufacturing
activities at the facility. HMWMD sent an inspection notification letter and information packet to all
group 3 SQGs about 30 days before it conducted its first group 3 inspections.
Task 4: Contact and Inspect Compass SQGs
In early 2001,HMWMD believed that there were about 1,200 SQGs in Colorado. As HMWMD began
contacting group 2 and 3 SQGs regarding upcoming inspections,it found that many of the facilities listed
in HMWMD's database as SQGs were actually conditionally exempt SQGs(CESQGs). Many regulatory
requirements that apply to SQGs do not apply to CESQGs. Therefore, HMWMD removed all CESQGs
from the SQG Compass project because the compliance information vital to the project objective could
not be collected from CESQGs. Removing the CESQGs decreased the number of facilities in the sample
size. To offset this problem,HMWMD added another sector(printers and publishers)to increase the
sample size. When HMWMD added the printing and publishing sector,it included enough SQGs to raise
the sample size back to 60 SQGs. However,many facilities in this sector listed in HMWMD's database
as SQGs were also actually CESQGs. Therefore,at its conclusion,the SQG Compass project included 49
SQGs(11 lower than HMWMD's intended sample size of 60). Groups 1, 2,and 3 included 25, 13, and
11 SQGs,respectively. Group 1 was larger than the other two groups because all SQGs with recent
inspections were assigned to group 1 as explained in the Task 2 description above.
Beginning later in 2001, HMWMD conducted group 1 inspections and prepared the pollution prevention
materials for group 3. HMWMD inspected group 2 SQGs from late 2001 through early 2002. HMWMD
inspected group 3 SQGs in mid 2002.
Task 5: Collect and Analyze Data
HMWMD conducted each inspection,regardless of the SQG's group assignment,as a normal compliance
evaluation inspection(CEI)with no immunity from any appropriate enforcement. The inspectors filled
out a Notice of Inspection(NOD form with the enhanced number(39)of regulatory coverage areas. For
group 2 facilities,inspectors also collected additional general information about the SQG(see data
collection sheet in Appendix B-1);for group 3 facilities inspectors collected the same general information
as for group 2 facilities as well as information about the SQGs use of the compliance assistance and
pollution prevention material(see data collection sheet in Appendix B-1).
Final Report 3-5 SQG Compass Project
3.3 DATA ANALYSIS PROCEDURE
HMWMD analyzed data from the three SQG groups quantitatively using a statistical methodology,and
descriptively using charts and graphs,to visually represent project results. This section describes
HMWMD's statistical data analysis procedure. The raw data are in Appendix B-2 and the statistical and
graphical results are presented in Section 3.4 and Appendices B-3 and B-4. The statistical analysis for
SQGs addressed two questions that are described in this section.
Question 1: Is there a difference in the average frequency of compliance per facility among the
three groups?
Note: a coverage area may encompass one or several closely related hazardous waste
regulations. An SQG must be in compliance with all requirements within a coverage area to
be considered in compliance.
Statement of Null and Alternative Hypotheses:
The hypothesis tests are structured in two tiers.
Tier 1: The first tier tests whether there is an overall difference among the three groups.
Ho: The average frequencies of compliant responses per facility in groups 1,2,and 3 are equal.
HA: The average frequencies of compliant responses per facility are significantly different
between at least two of the groups.
Tier 2: The second tier is contingent on rejecting Ho in the first tier(that is,concluding that at
least two groups are different). If Ho is rejected,pair-wise testing of all pairs of groups is
performed.
Ho: The average frequencies of compliance responses in groups X and Y are equal.
HA: The average frequencies of compliant responses are significantly different between groups X
and Y.
Where: X and Y refer to an individual pair of groups(for example, group 1 and 2, group 1 and 3,
group 2 and 3).
HMWMD addressed the first tier for Question 1 by comparing the average(or median)frequencies of
compliant responses per facility for each of the three groups using both a parametric(analysis of variance)
and nonparametric(Kruskal-Wallis test)test suitable for multiple samples. The results of this test are
provided in Section 3.4,Table 3-2. Because no significant difference among groups was shown in the
fast tier analysis,the second tier post hoc comparison of the three means was not conducted using the
Tukey-Kramer HSD test.
Final Report 3-6 SQG Compass Project
Question 2: Is there an overall difference in compliance rates for individual coverage areas
between the two"treatment"or"test" groups and the"control" group? [Stated
another way,this also addresses the question of whether the frequency of coverage
area violations is higher in the control versus the two treatment groups]
Note: Compliance rate is defined as the number of compliant responses divided by the
compliant responses plus the non-compliant responses.
Statement of Null and Alternative Hypotheses:
The approach is based on an aggregate response,where the individual responses for groups of
indicators within individual coverage areas(or for all indicators combined into a single group)are
combined. The hypothesis tests are structured in two tiers.
Tier 1: The first tier tests whether there is an overall difference among the three groups.
H,: The rates of compliance in groups 1,2,and 3 are equal.
HA: The rate of compliance is significantly different between at least two of the groups.
Tier 2: The second tier is contingent on rejecting Ho in the first tier(that is, concluding that at
least two groups are different). If Ho is rejected,pair-wise testing of all pairs of groups is
performed.
H,: The rates of compliance in groups X and Y are equal.
HA: The rate of compliance is significantly different between groups X and Y.
Where: X and Y refer to an individual pair of groups(for example, group 1 and 2,group 1 and 3,
group 2 and 3).
HMWMD evaluated the Question 2,Tier 1 hypothesis by arranging the data in a 2 x 3 contingency table
(columns were groups and rows were the counts of facilities where compliance for an individual coverage
area was reported as"Yes"or"No")and conducting a chi-square analysis. For cases where Ho was
rejected and it was concluded that at least two of the groups were different,HMV/MD conducted a post-
hoc pair-wise comparison of the individual pairs of groups. This analysis was performed on angular-
transformed proportions,calculated as:
p' =2I aresin nX lI +aresin n+1 J'
where: X and p'are the original and transformed proportions,respectively.
Final Report 3-7 SQG Compass Project
HMWMD then performed individual pair-wise comparisons using a Tukey-type multiple comparison test
modified for proportions,as described in Zar(1996)'. The results of tests performed for nine coverage
areas and among all coverage areas are provided in Section 3.4,Table 3-3,and in Appendix B-3.
3.4 DATA ANALYSIS RESULTS
This section summarizes the statistical data analysis results. All of the following results were analyzed
using a nominal confidence level of 95 percent. This means that one can conclude with at least 95 percent
confidence that the results of the statistical tests cannot be attributed to chance alone.
Question 1 asked whether there was a statistically-significant difference in the average compliance rate
per facility between the three groups. The results show that the average compliance rates were higher in
groups 2 and 3 compared to group 1,but that none of the differences found are statistically significant.
Table 3-2 presents the statistical analysis results.
TABLE 3-2
COMPARISON OF THE AVERAGE COMPLIANCE RATES
PER FACILITY AMONG ALL GROUPS
Average Frequency of Multiple Sample Comparison of
Compliant Responses per Group Means(Medians)
Facility for each Group' parametric Non Parametric
(Percent) (ANOVA) (Kraals"- Prob> Overall Conclusion
Prob>F Wallis) Chi-
t 2 3 F Ratio Chi-Square Square
(n=23) (n=13) (n=11)
59 77 71 0.984 0.382 0.503 0.778 M1=M2=M3
Notes:
See Section 3.3 for overview of statistical testing.
Group 1 is the control,groups 2 and 3 are treatment groups.
2 Multiple-sample parametric(ANOVA)and non-parametric(Kruskal-Wallis)tests were used to test the H0 that
the average frequency of compliance per facility is the same for all groups. ANOVA tests mean responses and
Kuskal-Wallis tests median responses. Probabilities of achieving either a greater F ratio or chi-square result
less than 0.05(5 percent)are considered statistically significant and 1-10 is rejected.
ANOVA Analysis of variance
H0 Null hypothesis
Prob Probability
Ml,M2,M3 Means or medians for groups 1,2,and 3,respectively
'Zar,J.H. 1996. Biostatistical Analysis. Third Edition. Prentice Hall. Upper Saddle River,New Jersey.
Final Report 3-8 SQG Compass Project
Data analysis for Question 2 began with a qualitative analysis to identify coverage areas for statistical
testing. HMWMD graphed the percent compliance for each group(1,2,and 3)in each coverage area;
Figure 3-1 is an example graph. Similar figures for each coverage area are included in Appendix B-4.
FIGURE 3-1
GROUP COMPLIANCE COMPARISON FOR COVERAGE AREA"PTG"
(Improperly Closed Containers of Hazardous Waste)
100
0
80 cd
El I : 10 facilities
60 8 ■ Crap 2:9faalities
40 Gulp 3:7 failities
20 N
Pitt lb' 0
Based on the results of the graphs,HMWMD selected nine specific coverage areas(GRA, GRB,PTFa,
PTG, PTJ,PTK,PTM, PTP,and PTQ)to statistically analyze(Question 2,Tier 1). HMWMD selected
these coverage areas for statistical analysis because either(1)their graphs each showed an increase in
compliance from group 1 to group 2 to group 3 or(2)the coverage area acts as an indicator of other
problems. For each instance when the null hypothesis was rejected,HMWMD evaluated Question 2,Tier
2. Table 3-3 summarizes the results of the analysis and Appendix B-3,Table B-3-1 includes more
detailed test results. Refer to section 3.3 for a more complete explanation of the Question 2 statistical
analysis.
Final Report 3-9 SQG Compass Project
TABLE 3-3
PROPORTION OF COMPLIANT FACILITIES AMONG GROUPS
Coverage Area Coverage Area Question 2,Tier Question 2 Tier
Code Description 1 Results 2 Results
GRA Failure to make an adequate hazardous waste Fail to Reject Ho P1=P2=P3
determination
GRB Failure to notify HMWMD of hazardous Reject Ho P1=P2=P3
waste activity
PTFa Failure to conduct weekly inspections of Fail to Reject Ho P1=P2=P3
hazardous waste storage containers
PTG Improperly closed hazardous waste Reject Ho P1=P2=P3
containers
PTJ Improper container labeling Fail Reject Ho P1=P2=P3
PTK Improper preparedness and prevention Fail to Reject Ho P1=P2=P3
activities
PTM Inadequate employee training Fail to Reject Ho P1=P2=P3
PTP Inadequate emergency response activities Reject Ho Pl<P3,PI=P2,
P2<P3
PTQ Inadequate SQG training Fail to Reject Ho P1=P2=P3
All above Combination of GRA,GRB,PTFa,PTG, Reject Ho PI<P2,PI<P3,
coverage areas PTT,PTK,PTM,PTP,and PTQ P2=P3
All 39 coverage Reject Ho PI<P2,PI<P3,
areas P2=P3
Notes:
�—. See Section 3.3 for overview of statistical testing
Ho Null hypothesis
P1,P2,P3 Proportion of compliant facilities in groups 1,2,and 3,respectively
Fail to Reject Ho: Conclude that the rates of compliance for groups 1,2,and 3 are statistically equal
Reject Ho: Conclude that there is a statistically significant difference in the rates of compliance for at least two
groups.
• Note that for the two tests where coverage areas were combined,the proportions shown were calculated by
dividing the total number of compliant responses by the total number of possible responses. This is not
equivalent to the proportion of compliant facilities with in each group,but rather is an overall measure of
compliance taking all facilities and coverage areas into account.
All of the individual coverage area results presented in Table 3-3 are suspect because the sample size was
low,which produces a statistically-biased result. Specifically,in a contingency analysis,one requirement
of the test is that at least 20 percent of the cells in the contingency table have values greater than 5.
Twenty percent of the contingency table cell values for SQGs were less than 5 causing the results to be
`suspect.' The fact that the results are suspect means that the results of the analysis are inconclusive. In
part,sample size was low because the number of SQGs in the project was lower than HMWMD intended
in its design. However,perhaps a more significant issue that affected the amount of data available for
evaluation was that the data set only contained a small amount of coverage area violation data. The data
HMWMD collected consisted of a spreadsheet organized by facility and by coverage area. Therefore,in
concept,it had data regarding whether each facility was in or out of compliance with each coverage area.
Final Report 3-10 SQG Compass Project
In practice,however,not all coverage areas applied to all SQGs and,in some cases, although the coverage
area applied to the SQG,no responses were available on the day of the inspection(for example,an
inspector might not be able to answer questions related to container condition if the SQG had recently
sent out a shipment of hazardous waste and had no hazardous waste at the time of the inspection).
Consequently,much of the coverage area data were not available for each SQG. This affected the
strength of the conclusions that could be drawn which were based on the assumptions that(1)all coverage
areas applied to all SQGs and(2)the data would be complete(that is,the data set would include an
indicator that a facility was either in our out of compliance with each coverage area).
HMWMD also analyzed the group of all 39 coverage areas as a whole to compare the proportion of
compliant facilities. The result of this test is highly significant(p<0.001)and the result is not qualified
(that is,there are a sufficient number of observations for each cell of the contingency table). The results
of this analysis indicate that when all facilities and all 39 coverage areas are considered together,there is
a statistically significant increase in compliance between group 1 and both groups 2 and 3,but that the
difference in compliance between groups 2 and 3 is not statistically different(meaning that giving notice
of an impending inspection increases compliance,but that including compliance assistance and pollution
prevention material,does not further increase compliance). Based on this result,HMWMD can conclude
that providing notice of an upcoming inspection appears to have had an overall effect on compliance.
However,HMWMD notes that the survey design conditions were not optimal and may have biased the
results. Thus,HMWMD can conclude based on the statistically analysis that its Compass program
positively affected the compliance of the SQGs,with the added qualifier that the sample design was
suboptimal and some bias may have been introduced into the analyses.
In addition to the data HMWMD collected for the statistical analysis,it also asked each of the group 2 and
group 3 facilities for general facility information such as number of employees,number of hazardous
waste satellite accumulation points,and whether the facility had an Environmental Manager. Table 3-4
summarizes the group 2 and group 3 responses to the general facility questions. Table 3-5 summarizes
the results of the remainder of the questions.
Final Report 3-11 SQG Compass Project
TABLE 3-4
SUMMARY OF PERCENT"YES"RESPONSES TO GENERAL QUESTIONS
Question Group 11 Group 21 Group 3' Overall'
Environmental management system in place? 4% 25% 0% 8%
Compliance assurance program in place? 24% 37% 36% 37%
Have designated environmental coordinator? 72% 92% 100% 83%
I Percent"Yes"responses calculated by dividing the total number of yes responses to the total number of yes
plus no responses.
TABLE 3-5
SQG QUALITATIVE DATA SUMMARY
QuestionGroup 1 Group 2 Group 3
Range Average Range Average Range Average
Minutes spent delivering 0 to 60 10.2 0 to 48 3.6 0 to 7.5 4.8
compliance assistance and
pollution prevention
material
Number of satellite 0 to 15 2.75 0 to 8 2.1 0 to 7 2.2
accumulation points
Number of 180-day 1 to 2 1.1 0 to 2 1 0 to 3 1.1
accumulation storage areas
Number of employees 4 to 275 69 28 to 300 134 5 to 300 134
HMWMD also asked group 3 facilities a series of questions related to the compliance assistance and
pollution prevention materials provided before the inspection. Table 3-6 summarizes group 3 responses
to questions related to the compliance assistance and pollution prevention materials.
TABLE 3-6
GROUP 3 ONLY QUESTION RESPONSES
Question Group 3
Percent Yes
Read compliance assistance material? 63%
Read pollution prevention material? 54%
Have questions or comments about 27%
compliance assistance material?
Have questions about pollution prevention 0%
material?
Interested in P2 follow-up? 40%
SQG contact HMWMD before inspection? 0%
Percent"Yes"responses are calculated by dividing the total number of yes responses to the total number of
yes plus no responses.
Final Report 3-12 SQG Compass Project
3.5 SUMMARY OF FINDINGS
This section summarizes the statistical and descriptive findings of the SQG Compass project.
3.5.1 Statistical Analysis of Compliance Data
HMWMD conducted a statistical analysis of the compliance data to answer the questions presented in
Section 3.3. HMWMD made the following determinations and deductions from the statistical results:
1. The first statistical test HMWMD conducted(Question I)evaluated the overall compliance rates
per facility between groups 1,2,and 3. The results show that the average compliance rates were
higher in groups 2 and 3 compared to group 1,but that none of the differences found are
statistically significant.
2. The second statistical evaluation(Question 2,Tier 1)found three instances(GRB,PTG, and PTP)
of statistically-significant differences among groups at the coverage-area level,which were then
evaluated at the Tier 2 level. Two of the Tier 2 tests(coverage areas ORB and PTG)led to
contradictory results(that is,they did not confirm the conclusion of the Tier 1 test that at least
two of the groups are different). This is not surprising,given that the results of the Tier 1 testing
were suspect because of low sample-sizes,and because each tier of testing was based on different
statistical tests. For PTP,significant differences were shown between groups 1 and 3 and groups
2 and 3,but not between groups 1 and 2. Again,in all of the instances where the Question 2 null
hypothesis was rejected under Tier 1 (leading to the conclusion that there is a significant
difference among at least two groups),results are suspect (refer to Section 3.3 for further
description of the statistical analysis and Section 3.4 for a description of the results and more
detailed explanation of suspect statistical results).
3. HMV/MD also evaluated the overall compliance rates taking all facilities and all 39 coverage
areas into account. The results for this analysis indicate a statistically significant difference
between group 1 and both the group 2 and 3 SQGs,but no significant difference was shown
between groups 2 and 3.
4. Based on these analyses,HMWMD concluded that the Compass project shows that inspection
notification has a net positive effect on SQG compliance rates. While small sample-sizes and
other limitations of the survey design did not allow for an optimal test of the effectiveness of
notification at the level of an individual facility or coverage area,when the results for all facilities
and coverage areas are evaluated together,the overall effect of notification is clearly shown. The
two levels of notification were shown to be approximately equivalent in their effectiveness,
suggesting that there is no added benefit to implementing the higher level of pre-inspection
notification.
5. Before conducting the statistical analysis,HMWMD plotted the group 1,2,and 3 compliance for
each of the 39 coverage areas. The graphs are located in Appendix B-4. For 23 coverage areas,
compliance either remained the same,decreased,or showed variation among groups(for
example, group 1 =30 percent,group 2=25 percent,and group 3=47 percent). Table 3-7
summarizes the coverage areas in each of these three categories.
Final Report 3-13 SQG Compass Project
0 TABLE 3-7
SELECT COVERAGE AREA PERFORMANCE SUMMARY
No Change in Compliance Decrease in Compliance Variable Compliance
CEG CESQG GLB Land ban requirements MRA General manifest
requirements PTA Pre-transport requirements
MRB Number of manifest packaging GOR Other generator requirements
copies PTB PUC/DOT labeling MRC Use of the manifest
PTH Incompatible wastes PTC Pre-transport marking PTI Tank management
PTKb Pre-transport pTD Placarding PTO Satellite accumulation
requirements: PTT, Contingency plan PTM Pre-transport training
preparedness and
prevention FIR Other pre transport PTN Waste analysis plan
requirements RRA Record keeping
RRB Biennial reporting
UWR Universal waste requirements
6. HMWMD attributes both the relatively low number of responses and,consequently,potentially
biased statistical results,to four primary factors described below.
a. Survey Design. More significant consultation with a statistician during the planning phase
for the SQG Compass project would have improved the project design. For example,a
statistician would have been able to give advice regarding the selection of random samples,
establishing appropriate sample sizes,specifying a desired level of statistical power for
selected tests, and providing advice for handling difficulties in the facility selection process,
recording data,and managing the data in a database.
b. Direction from EPA. EPA advised HMWMD to only include sectors in the SQG Compass
project that were small enough that HMWMD could include the entire population of the
sector in the project. HMWMD followed this advice,but found that it significantly limited
the sector choices for the project and ultimately contributed to the insufficient sample size.
c. Incorrectly Classified Facilities in HMWMD Database. Many facilities were incorrectly
labeled in HMWMD's database as SQGs,but were really CESQGs. `Over-notification' by a
CESQG is not illegal and many companies over-notify to be conservative in filing. However,
HMWMD inspects hazardous waste generators according to how they are operating on the
day of the inspection,regardless of how they have notified. Therefore, since CESQGs are
subject to fewer regulations,they were not appropriate for inclusion in the SQG Compass
project. HMWMD added a sector to compensate for the decreased number of facilities when
CESQGs were removed;however, it was not able to attain its goal of 60 SQGs(20 in each
group)for the project.
d. Inspection Data Collection. Not all coverage areas applied to all SQGs and,in some cases,
inspectors were unable to obtain information for some coverage areas. Refer to Section 3.4
for further discussion of this issue. Ultimately,the amount of data was less than it otherwise
would have been,which affected the quantity of data available for conducting statistical tests
using contingency tables.
3.5.2 Descriptive Analysis
Although no significant statistical significance is associated with the descriptive data collected from each
facility,some interesting deductions can be made:
Final Report 3-14 SQG Compass Project
1. By expanding the number of coverage areas addressed on the NOI form,HMWMD has been able
to more accurately identify compliance related issues on all hazardous waste inspections,not just
those included in the SQG Compass project. For example, HMWMD can now distinguish
between illegal storage,disposal,and treatment of hazardous waste;when HMWMD had only 7
coverage areas,these violations were included in one coverage area. HMWMD found the
additional information to be very useful and continues to record violation data from inspections
using the 39 coverage areas.
2. Seven of the 11 group 3 facilities indicated that they had read the compliance assistance
information provided before the inspection. However,the group 3 compliance rate was not
notably improved over either the group 1 or group 2 compliance rates. HMWMD concludes that
either:
a) facilities over-estimated their efforts to read the materials sent to them,
b) facilities made changes,but had difficulty translating the guidance into improvements
in their compliance,and/or
c) facilities read the information,but did not pay any attention to it.
3. Based on the graphs of the percent compliance for each coverage area(located in Appendix B-4),
using no statistical analysis,four coverage areas showed a noticeable increase in compliance from
group 1 to group 2 to group 3. The coverage areas showing consistent improvement are
summarized in Table 3-8,below.
TABLE 3-8
GRAPHICALLY NOTICEABLE COMPLIANCE IMPROVEMENT BY COVERAGE AREA
Coverage Group 1 Group 2 Group 3
Areal Percent Percent Percent
Compliance Compliance Compliance
PTG 20 67 86
PTJ 20 55 67
PTK 45 83 100
PTP 14 36 100
'See Table 3-3 for coverage areas descriptions
4. More than half of the group 3 SQGs indicated that they read the compliance assistance and
pollution prevention materials. Because of the sample size,HMWMD cannot make any
statistically-based conclusions regarding the effect its efforts had on SQGs compliance
performance. HMWMD should reevaluate the study using a statistically defensible survey design
and draw conclusions from those results.
5. HMWMD asked the questions listed in Table 3-4 to reinforce the statistical fmdings and augment
its understanding of facility activities that affect compliance. Several interesting deductions can
be made from this information:
a. Very few(4 out of 49)facilities claimed to have implemented an EMS. Of those that did,
compliance ranged from 80 percent on the low end to 100 percent on the high end. Overall,
these facilities had an average compliance rate of 91 percent,much higher than the 73 percent
overall compliance rates across all three groups. However, so few SQGs have EMSs that
HMWMD placed little weight on this conclusion.
Final Report 3-15 SQG Compass Project
b. There seems to be low correlation between compliance rates and the presence of a
compliance assurance program at a facility. From Figure 3-2,which includes data from all
three groups of facilities,the two bars represent SQGs with and without compliance
assurance coordinators. 84 percent of the SQGs with compliance assurance programs have
compliance greater than 60 percent compared with 68 percent of the SQGs without
compliance assurance programs. However,percentage wise,the number of facilities with
compliance greater than 80 percent is about equal among SQGs with and without compliance
assurance programs.
c. There may be a correlation between compliance rates and having a designated environmental
manager at a facility. Figure 3-3 is set up in the same manner as Figure 3-2,but shows the
presence or absence of an environmental coordinator or manager at the site compared to
percent compliance. Here,75 percent of the SQGs with an environmental coordinator have
compliance above 60 percent and 52 percent above 80 percent compared 43 percent above
both 60 and 80 percent compliance for SQGs with no environmental coordinator. However,
22 percent of the SQGs with environmental coordinators have compliance below 20 percent,
which decreases the strength of the conclusion that having and environmental coordinator
increases the compliance of an SQG.
Final Report 3-16 SQG Compass Project
FIGURE 3-2
COMPLIANCE ASSURANCE PROGRAM
COMPARISON WITH COMPLIANCE
16
•Comptiace n.em.nce Program 14
td' •No Compliance A nice Program
12-
Fj . 0
0
■
0
0 to 20 20 to 40 40 to 60 6060 80 80 to 100
Percent Compliance(%)
FIGURE 3-3
DESIGNATED ENVIRONMENTAL COORDINATOR
COMPARISON WITH COMPLIANCE
25
■environmonS Coordinator
ONo Environmental Coordimtor 21
20
15'
rom..
E •10' 0 9
Z
- 0 • 0 - 0
•
0 to 20 20 to 40 406060 606080 80to 100
Permit Compliance(%)
6. HMWMD inspectors recorded two types of information related to pollution prevention during
the inspection of the Compass SQGs. The first was a brief description of any waste
minimization and pollution prevention opportunities discovered or discussed with the SQG.
HMWMD included this item to stimulate the inspector to look for opportunities for waste
minimization or pollution prevention,and discuss those opportunities during the inspection, (as
opposed to looking for past pollution prevention efforts by the SQG.) Inspectors identified
pollution prevention opportunities at 16 of the 49 facilities. HMWMD does not know whether
these facilities implemented any of the suggested pollution prevention opportunities.
Secondly,inspectors were asked to indicate whether any group 3 SQGs were interested in
pollution prevention follow-up,such as a pollution prevention assessment or energy audit or
further information based on the pollution prevention fact sheets they had received before the
inspection. Of the group 3 SQGs,four facilities indicated some interest,but the interest was
limited to either pollution prevention or energy efficiency audits. Based on the technical content
of the pollution prevention materials provided to the SQGs,HMWMD was disappointed by this
lackluster response.
Final Report 3-17 SQG Compass Project
7. Table 3-5 presents several additional questions HMWMD inspectors asked each SQG. The
results of evaluating this data is presented below:
a. HMWMD tracked the time spent delivering compliance assistance before and during the
inspections of all the facilities to determine a cost-benefit analysis of the compliance rate
improvements. However,because statistical results indicated no statistically significant
difference in compliance between groups 2 and 3,HMWMD did not evaluate the cost-benefit
of the two levels of intervention.
b. HMWMD could not establish a clear relationship between compliance rates and 1)the
number of waste streams generated at a facility,2)the number of employees at a facility,or
3)the number of satellite accumulation points at a facility. Further, HMWMD could not
establish a clear relationship between the number of employees and those facilities that had
designated environmental coordinators.
Final Report 3-18 SQG Compass Project
4.0 CHROME PLATING FACILITIES
RESULTS AT A GLANCE
The Colorado Department of Public Health and Environment(CDPHE)Air Pollution Control Division
(APCD)divided all 29 chrome plating facilities in Colorado into three groups. Each group received
different levels of inspection notification and compliance assistance before APCD conducted an
inspection. During the inspections,APCD noted all violations. APCD conducted two statistical tests
on the data it collected during the inspections. Overall the statistical tests showed no significant
difference in the average frequency of compliance per facility among the three groups,and found only
one instance of a statistically significant difference among groups when tests were conducted for
individual indicators. However,all of the test results for individual indicator questions are suspect
because the sample sizes were too low to allow for unbiased statistical comparisons. Therefore,the
only general conclusion APCD can draw is that the statistical analysis results are inconclusive. APCD
attributes the low sample-sizes primarily to the manner in which the survey was designed and the fact
that some data were collected,but not included in the analysis.
4.1 PROJECT DESCRIPTION
The APCD chrome plating Compass project involved dividing the 29 Colorado chrome plating facilities
(note that 31 inspection forms were filled out because two chrome platers had two different types of
plating operations)into three groups,providing different levels of compliance and pollution prevention
assistance to each group,and comparing the compliance of the groups. APCD chose the chrome plating
facilities for Compass for the following three reasons:
5. Unsatisfactory Compliance Performance: In the past,APCD perceived the overall compliance
of asphalt manufacturers to be unsatisfactory.
6. Potential for Negative Impact on Human Health and Environment: The chrome platers in
Colorado tend to be located within heavily populated areas and have the potential for causing
negative impact on human health and the environment if the facilities are improperly managed.
7. Consistent Sector in Terms of Equipment and Technology: The chrome plating industry in
Colorado,with a few exceptions,consists of small businesses with relatively similar equipment
and technology(including environmental control technology).
4.2 PROJECT IMPLEMENTATION APPROACH
An overview of the chrome plating Compass project milestones and schedule are presented in Figure 4-1
and discussed in further detail in the sections following the timeline.
Final Report 4-1 Chrome Plating Compass Project
FIGURE 4-1 CHROME PLATING COMPASS PROJECT TIMELINE
Task 1:Create inspection Task 3: Select,contact,and inspect
checklists(12/00)
chrome plating facilities(6/02)
I � I
Task 2:Write pollution
prevention fact sheet for chrome
plating facilities(12/00)
Task 1: Create Inspection Checklists
APCD reviewed its existing inspection checklists for chrome plating facilities and modified the questions
so that it could easily use the data it collected for statistical evaluation. APCD has four different versions
of inspection checklists to account for different types of plating operations. The inspection checklists
were edited so that:
• Questions were worded identically on each inspection checklist.
• All questions necessary for statistical analysis were included on all checklists.
• All compliance questions were worded such that a binomial(yes/no)response would be evoked.
• All questions were worded such that a"yes"response indicated compliance and"no" indicated
noncompliance.
Task 2: Write Pollution Prevention Fact Sheet for Chrome Plating Facilities
APCD created a pollution prevention fact sheet for chrome plating facilities that discusses dragout
reduction and water saving opportunities for chrome plating facilities. The fact sheet also includes a
checklist that addresses common pollution prevention opportunities at metal finishing facilities. The
main purpose of the checklist is to give chrome plating facilities the opportunity to assess the extent to
which they have implemented pollution prevention opportunities in their shops. APCD sent the fact
sheets to a subset(group 3)of chrome plating facilities involved in Compass(see Task 3 and Table 4-1
for further explanation.)
Task 3: Select,Contact,and Inspect Compass Chrome Plating Facilities
APCD selected 29 chrome platen in Colorado,representing 100 percent of the sector in Colorado.
Although there are 29 chrome platen in Colorado,APCD competed two more inspection forms than
inspections because two chrome platers had two different types of plating operations. The locations of
the chrome plating facilities are shown in Figure 4-2. APCD divided these 29 facilities into three
randomly-selected groups with each group receiving a different amount of compliance assistance as
explained in Table 4-1.
Final Report 4-2 Chrome Plating Compass Project
ii �� ) Sen9 k
Logan
Jackson Larhner
fdo=fal f ` I Pmllips
d
RDlli( `----.. �— r
"u1.[inn
� Gland 3outde
?iu Blares) — — *--- I Washington Yuma
I� -� L`anve. Adams
I
Carried �c --.};lear C NP, AraOahoe
tagl,
J nrs
S Untf.--fii P'
Lr / ` Kit Carson
_ Wtkirl Lake If glas s
( Parr
-�` Lincoln '
Mesa Deity -- `� T,ir.. F.Pa:;c
�� �� �_r Cheyenne
! I
G❑nl ' -son C Gh-tP :� _ �
V
F mn:ant
1 Kin
no:.:rose ' N ws
` ` i Cm nIPy I
-- Ouray ), � 1 Fuel �—_
Sagurychs Custer
an 1,1 gaolHinsdale \
�\ P
J-✓ Il \.r.,./\-,, V_ _� r,p.r, Bent rowers
Clore. ' San Joan �� ��
Mineral I
f o r3Tarco Alamosa
7 1
�/� lns Anama5 Baca
A it ssa-na /
ostfi a
La 'Iola Ancooleta �I Goners);
I
1 C
0 25 50 100 150 200
Legend Miles
Chrome Plating Location N
Interstate Highway FIGURE 4-2
County Boundary
CHROME PLATING
COMPASS PROJECT
TABLE 4-1
CHROME PLATER GROUP ASSIGNMENTS
Group Number d Description
Facilities
1 9 "Control group" APCD notified chrome plating facilities of
inspections in the historically typical manner;that is,no prior contact
with the facility other than the call scheduling the inspection 5 to 10
working days in advance.
2 7 APCD sent a letter in advance of the inspection notifying the facilities
that APCD would conduct an inspection to evaluate compliance with
applicable federal Maximum Achievable Control Technology and
state air regulations at the facility sometime within 9 months.
3 APCD sent the same letter to Group 3 as it sent to Group 2. In
addition,APCD sent several compliance guidance documents and a
pollution prevention fact sheet relevant to the chrome plating
operations.
Note that CDPHE intended to inspect all 29 Colorado chrome platers for the chrome plating Compass
project,but data for analysis were only available for 21 of those facilities(and 23 inspection forms
because two chrome platers had two different plating operations). Some facilities closed before the
inspections occurred and certain inspectors completed incorrect inspection forms,resulting in data that
were not useful for the Compass project. From June 2001 through June 2002,APCD inspected each
Compass chrome plating facility that was still in business and compiled the compliance results. The data
analysis procedure and results are presented in Sections 4.3 and 4.4.
4.3 DATA ANALYSIS PROCEDURE
APCD intended to inspect all 29 Colorado hard and decorative chrome plating facilities;however,as
explained in Section 4.2,Task 3, some facilities closed before the inspection and APCD used incorrect
inspection forms for some of the inspections. As a result, APCD could only analyze data from 21 of the
chrome plating facilities(and 23 inspection forms). A further nuance of the chrome plating facility data
analysis was that APCD asked each facility questions from one of four inspection checklists,according to
the type of chrome plating each facility performed. Each checklist had questions in common with other
checklists and questions that differed from other checklists. For the sake of direct comparison and
requirements of the statistical analysis,only questions that could be answered with a simple yes or no
response and were common to all four checklists were included in the analysis';28 questions met this
requirement. Therefore,APCD's quantitative analysis using formal statistical tests and descriptive
'Eliminating yes/no questions that were not common to all inspection forms after the inspections were complete
introduced unknown bias into the statistical analysis. The four inspection checklists included between 39 and 78
yes/no compliance related questions.
Final Report 4-4 Chrome Plating Compass Project
analysis using charts and graphs to visually represent project results,were based on data from the 28
questions in common with all four checklists from the 23 chrome plating inspection forms for which
APCD had inspection data. This section describes APCD's statistical data analysis procedure. The
statistical results are presented in Section 4.4 and Appendix C-1. The key to match indicator numbers
with the questions they represent is in Appendix C-2. The statistical analysis for chrome plating facilities
addressed two questions that are described in this section.
Question 1: Is there a difference in the average compliance rates per facility among the three
groups?
Statement of Null and Alternative Hypotheses:
The hypothesis tests are structured in two tiers.
Tier 1: The first tier tests whether there is an overall difference among the three groups.
H,: The average compliance rates per facility in groups 1,2,and 3 are equal.
HA: The average compliance rates per facility are significantly different between at least two of
the groups.
Tier 2: The second tier is contingent on rejecting Ho in the first tier(that is,concluding that at
least two groups are different). If Ho is rejected,then all pairs of groups were tested.
Ho: The average compliance rates per facility in groups X and Y are equal.
HA: The average compliance rates per facility are significantly different between groups X and
Y.
Where X and Y refer to an individual pair of groups.
APCD addressed the first tier for Question 1 by comparing the average compliance rates per facility for
each of the three groups using both a parametric(analysis of variance)and a nonparametric(Kruskal-
Wallis)test suitable for multiple samples. The result for the first tier was not significant,therefore,
APCD did not complete the second tier of the analysis for Question 1. The results of this test are
provided in Section 4.4,Table 4-2.
r^�
Final Report 4-5 Chrome Plating Compass Project
Quesion 2: Is there an overall difference in compliance among the three groups(control and
two"treatments")with respect to each performance indicator?
Statement of Null and Alternative Hypotheses:
The hypothesis tests are structured in two tiers.
Tier 1: The first tier tests whether there is an overall difference among the three groups.
Ho: The total proportion of responses indicating compliance for indicator X is the same for the
three groups.
HA: The total proportion of responses indicating compliance for indicator X is not the same for
the three groups.
Tier 2: The second tier is contingent on rejecting Ho in the fast tier(that is,concluding that at
least two groups are different). If Ho is rejected,then pair-wise testing of all pairs of groups is
performed.
Ho: The rate of compliance in groups X and Y is equal.
HA: The rate of compliance is significantly different between groups X and Y.
Where X and Y refer to an individual pair of groups.
Question 2 for chrome plating facilities was addressed using the same approach described for Question 2
for SQGs(see Section 3.3). The only difference is that the analysis for chrome plating facilities focused
on differences in the proportion of facilities showing compliance for individual performance indicators.
The first tier analysis was conducted using the same 2 X 3 contingency table approach,and the second
tier was conducted using a Tukey-type post hoc pair-wise comparison of the individual pairs of groups.
In this case,both tiers of tests were performed. The results for this analysis are provided in Appendix C-
1,Table C-1-1.
4.4 DATA ANALYSIS RESULTS
The test to determine if there was a statistically significant difference in the average compliance rates per
facility among the three groups(Question 1)failed to reject the null hypothesis,leading to the conclusion
that the average compliance rate per facility is the same for all three groups-. Table 4-2 presents the
statistical analysis results.
The operations at chrome plating facilities differ in(1)the type of chrome plating operations performed at
the facility and(2)the methods the facility used to control emissions from chrome plating operations.
r^.
Final Report 4-6 Chrome Plating Compass Project
There were four types of chrome plating operations among the facilities in the chrome plating Compass
project as follows:
• Decorative chrome plating tank using a trivalent or hexavalent solution,with a wetting agent as
an additive
• Decorative chrome plating tank using hexavalent solution with an add-on control device
• Hard chromium tank using an add-on control device
• Hard chrome plating tank using an add-on control device with a chemical fume suppressant
TABLE 4-2
COMPARISON OF THE AVERAGE COMPLIANCE RATES PER FACILITY AMONG GROUPS
Average Frequency Multiple-Sample Comparison of Group Means2
of Compliant
Responses per
Facility for Each Parametric Non-Parametric Prob>Chi- Overall Conclusion
Group (ANOVA) Prob>F (Kruskal-Wallis)
(Percent) Square
1 2 3 F Ratio Chi-Square
No Significant
52 47 59 1.100 0.352 3.901 0.142 Difference Among
Groups
Notes:
See Section 4.3 for full explanation of test details.
i Group 1 is the control,Groups 2 and 3 are treatment groups.
2 Multiple-sample parametric(ANOVA)and non-parametric(Kruskal-Wallis)tests were used to test the Ho that the
average compliance rates per facility are the same for all groups. Probabilities of achieving either a greater F ratio
or chi-square result less than 0.05(5 percent)are considered statistically significant and Ho is rejected.
ANOVA Analysis of variance
Ho Null hypothesis
Prob Probability
APCD has a different inspection foam for each of these four types of plating operations. Among these
forms,there were 28 indicators common to all four forms. The statistical analysis(Question 2)of each of
the 28 indicators showed that one indicator(indicator 21: Did the operator maintain records of the process
operating time for each tank and the date and time of each addition of the wetting agent?)showed a
statistically meaningful difference among groups I,2,and 3. Appendix C-1,Table C-1-I contains the
specific statistical analysis results for all 28 indicators for Tier 1 and 2 of Question 2. This finding,
however,is suspect because of the low sample-sizes used in the contingency analysis,which lead to
biased test results. Specifically,in a contingency analysis,one requirement of the test is that at least 20
percent of the cells in the contingency table have values greater than 5. More than 20 percent of the
contingency table cell values for chrome plating facilities were less than 5,causing the results for all tests
Final Report 4-7 Chrome Plating Compass Project
to be `suspect.' The outcome,therefore,is that all of the test results comparing compliance rates among
groups for individual indicator questions are inconclusive.
In addition to the statistical analysis,APCD graphed the group 1,2,and 3 results for each indicator. The
indicators were divided into six categories of violation as follows:
• Reports
• Operating and maintenance plans
• Record keeping
• Initial notification
• New tank installation
• Notification of compliance status
Figures 4-3 through 4-8 show the results for each category of regulatory violation for the groups1,2,and
3. Appendix C-3 contains a key to match indicator numbers with the questions they represent.
FIGURE 4-3
CHROME PLATING COMPLIANCE PERFORMANCE FOR REPORTING INDICATORS
loo
8 so
IE_i
E
1 2 3 4 5 6 7 8 9
Indicator
FIGURE 4-4
CHROME PLATING COMPLIANCE PERFORMANCE
FOR OPERATION AND MAINTENANCE PLAN INDICATORS
moll _
80
a
E - ■Group 1
U 40 _ ■Group 2
❑Group 3
1y 20
10 11 12 13 14 15 16
Indicator
Final Report 4-8 Chrome Plating Compass Project
FIGURE 4-5
CHROME PLATING COMPLIANCE PERFORMANCE
FOR RECORD KEEPING INDICATORS
100 00 mil 1 . ■ -
p I ■ ' • n ■Group 1
!" 6° 1 - ! , ' " .� ■Group 2
2
` e1 1 1 1 I - IIN O
r7 Group
17 18 19 20 21 22 23
Imitator
FIGURE 4-6 FIGURE 4-7
CHROME PLATING COMPLIANCE CHROME PLATING COMPLIANCE
PERFORMANCE PERFORMANCE
FOR INITIAL NOTIFICATION INDICATORS FOR NEW TANK INSTALLATION
100 bou100
W ■Group 1
560 ❑Group 2 a 80 ■Group 2
1.— _—a go —�Groff 1 c 60 O Group 3
6
6 6
U V
- 40 g 40
y 20 y 20
0 0
24 25 26
Imitator Indicator
FIGURE 4-8
CHROME PLATING COMPLIANCE
PERFORMANCE
FOR NOTIFICATION OF COMPLIANCE
STATUS INDICATORS
Note that the absence of a bar in Figures 4-6, a 100 ■Group 1
4-7,and 4-8 indicates that the group compliance - 80 •
4Croup 2
a. 60
with the indicator was 0 percent(for example, if) 40 Group 3
group 3 was in 0 percent compliance with d 20 16 IIIL
indicator 24). d 0
27 28
Indicator
T
Final Report 4-9 Chrome Plating Compass Project
4.5 SUMMARY OF FINDINGS
Based on Figures 4-3 through 4-8,one indicator(Indicator 18: "Did the operator maintain malfunction
records that document the malfunctions that occurred on the monitoring equipment?")showed a
noticeable increase in compliance from group 1 to group 2 to group 3,but the statistical analysis did not
show this increase to be statistically significant. It is unknown whether this increase would be statistically
significant given a larger sample-size. Therefore,because of the low samples-sizes,the test results are
inconclusive. Table 4-3 summarizes average percent compliance of each group and all groups combined
for the violation categories shown in Figures 4-3 through 4-8. (note that no statistical meaning can be
associated with the data in Table 4-3 because the data were not subjected to statistical analysis):
TABLE 4-3
AVERAGE PERCENT COMPLIANCE AMONG VIOLATION CATEGORIES
VIOLATION CATEGORY GROUP 1 GROUP 2 GROUP 3 ALL
GROUPS
Reporting(9 indicators) 36 41 25 34
Operating and maintenance(7 indicators) 55 39 69 54
Record keeping(7 indicators) 64 67 86 73
Initial notification(2 indicators) 7 8 8 5
New tank installation(1 indicator) 0 100 100 67
Notification of compliance status(2 indicators) 50 24 25 33
Overall the statistical tests showed no significant differences in the average compliance rates per facility
among groups(statistical Question 1),and found only one instance(indicator 21 —"Did the operator
maintain records of the process operating time for each tank and the date and time of each addition of the
wettig agent?")of a statistically -significant difference among groups(statistical Question 2)when tests
were run for individual indicator questions. This result,however,is suspect because there were not
enough data to allow for a statistically-unbiased comparison(refer to section 3.3 for further description of
the statistical analysis). In fact,all results of the statistical analysis are suspect because of low sample-
sizes. Thus,APCD must conclude that all findings from the statistical analysis are inconclusive. APCD
attributes both the relatively low sample-sizes and,consequently, statistically-biased comparisons,to
three primary factors as described below.
1. Survey Design. More significant consultation with a statistician during the planning phase
for the chrome plating Compass project would have allowed for the optimal design. For
example,a statistician would have been able to give advice regarding the selection of random
samples,establishing appropriate sample sizes,specifying a desired level of statistical power
for selected tests,and providing advice for handling difficulties in the facility selection
process,recording data,and managing the data in a database.
Final Report 4-10 Chrome Plating Compass Project
2. Missing Data. APCD originally intended to collect data for Compass project from all 31
Colorado chrome plating facilities. However,data from eight of these facilities(two group I,
two group 2,and four group 3)was not included in the quantitative(statistical) or qualitative
(graphical)analysis. This happened because several facilities closed before the inspections
occurred and certain inspectors completed incorrect inspection forms,resulting in data that
were not useful for the Compass project.
3. Common Indicators: Although the inspection forms for the chrome plating facilities
included from 39 to 78 yes/no questions,only 28 yes/no questions were common to all four
inspection forms used for the chrome plating facilities in this project. The limited number of
indicators not only decreased the number of indicators that were analyzed for the project,but
also introduced unknown bias into the statistical analysis results by not including all the data
collected in the analysis.
Final Report 4-11 Chrome Plating Compass Project
5.0 SURFACE WATER TREATMENT PLANTS
RESULTS AT A GLANCE
The Colorado Department of Public Health and Environment Water Quality Control Division
(WQCD)conducted comprehensive performance evaluations(CPE)at 28 surface water treatment
plants(SWTP)to identify performance limiting factors(PLF). PLFs are any direct or indirect aspects
of SWTP operation that can negatively affect the performance of the SWTP,and thus the treated
water quality. WQCD did not conduct a statistical analysis of the data it collected because neither
baseline data nor separate years of data were unavailable to use for comparison. The results of the
program,for the purpose of the Compass project,was an analysis of the number and type of PLFs
identified and the percent of the facilities that took action to eliminate those PLFs based on telephone
interviews with the SWTPs about two years after the CPE. The following table summarizes the
overall results of the CPE Compass project. The average percent action taken refers to the percent of
PLFs addressed subsequent to WQCD's identification of PLFs.
Range of PLFs
PLFs Identified Average Percent
PLF Group (All CPEs) Identified Action Taken
(Each CPE)
Level A and B PLFs 116 2 to 8 55
Level A 48 0 to 3 63
Level B 68 0 to 5 50
WQCD will continue this project by visiting each of the SWTPs to verify the information gathered in
the follow-up phone calls and will also extend the program to 41 other SWTPs in Colorado.
5.1 PROJECT DESCRIPTION
The Colorado Department of Public Health and Environment,Water Quality Control Division(WQCD)
Compass project involved conducting comprehensive performance evaluations(CPE)at 28 surface water
treatment plants(SWTP)that were failing to provide a minimum of three-log removal of particulate
matter as measured by the microscopic particulate analysis(MPA)test of untreated and treated water.
Historically,WQCD did not have resources to investigate possible treatment system failures,so these
potentially at risk situations received no formal attention. WQCD traditionally relied solely upon
reviewing turbidity measurements submitted monthly and also reviewed during routine on-site
inspections WQCD conducted about every 5 years. However,turbidity is generally very low in many
Colorado surface waters and thus, is not an accurate indicator of water quality. The Compass CPE project
provided compliance assistance to participating SWTPs in a manner that hopefully will avoid treatment
Final Report 5-1 CPE Compass Project
failures that could cause illness or death of consumers and obviate the need for use of submitted monthly
and also reviewed enforcement action and penalties.
WQCD designed the project to(1)identify causes of inadequate SWTP performance and(2)provide
facilities with a report that identifies and prioritizes SWTP performance limiting factors(PLF). WQCD
chose SWTPs for its Compass project for the following reasons:
1. Water supply treatment process requirements became more challenging for SWTPs to comply
with in recent years.
2. The performance of SWTPs can directly and acutely impact human health in a community.
3. Research supports the need for consistent particle removal by water treatment facilities to
maximize public health from microbial contamination.
5.2 PROJECT IMPLEMENTATION
An overview of the Compass CPE project milestones and schedule are presented in Figure 5-1 and
discussed in further detail the in sections following the timeline.
FIGURE 5-1
CPE COMPASS PROJECT TIMELINE
Task 1: Task 3: Task 4: Task 6:
Develop CPE project Conduct CPEs Write CPE report SFY Make CPE follow-up phone calls
SFY 2000 SFY 2001 2001 SFY 2003
SFY SFY
20001 2003
Task 2: Task 5: Task 7:
Select SWTPs for CPEs Send CPE reports to SWTPs Conduct CPE follow-up site visits
SFY 00-01 SFY 2001 (date dependent on iri-annual
sanitary survey schedule)
Task 1: Develop CPE Project
Using the EPA Handbook"Optimizing Water Treatment Plant Performance,"WQCD and a support
contractor designed the CPE Compass project to systematically evaluate SWTPs whose MPA results
indicated performance potentially inadequate to comply with the Surface Water Treatment Rule. The
Surface Water Treatment Rule establishes federal and Colorado treatment technique requirements for
removal or inactivation of Giardia,viruses,and more recently,cryptosporidium.
Task 2: Select SWTPs for CPEs
To identify appropriate treatment plants for CPEs,WQCD used the microscopic particulate analysis
(MPA)test as a screening method. The MPA test identifies surface water"bioindicators" such as plant
Final Report 5-2 CPE Compass Project
debris,algae,diatoms,insects,rotifers, Giardia,and coccidia that are commonly present in surface waters.
The MPA method involves filtering a minimum volume of both raw and treated surface water through a 1
micron(µm)nominal-pore-size fiber wound filter over a maximum of a 24-hour period. The filter is then
processed in the laboratory by eluting the particles from the fibers,concentrating the eluant,and
microscopically examining slides for bioindicators. The bioindicators are quantified and used to calculate
a removal percentage,which then can be used to help evaluate removal performance relative to the
Surface Water Treatment Rule requirements.
WQCD targeted SWTPs for its Compass project that had less than 3-log removal within at least 1-year of
previous monitoring. In 2000,the WQCD identified 69 facilities to participate in the Compass CPE
project, 28 of which WQCD selected for the first year of the program. The project was designed to help
water systems identify and correct issues that may limit their ability to achieve 3-log removal
performance.
WQCD hired a third party contractor to conduct CPEs at selected SWTPs. Figure 5-2 shows the
distribution of project participants throughout Colorado.
Task 3: Conduct CPEs
In State fiscal year(SFY)2001,the WQCD conducted on-site CPEs at 28 SWTPs. During the site visit,
the contractor worked with SWTP employees to assess the performance of various aspects of the facility
operation,including: design, instrumentation and automation,filtration, unit process adequacy,process
control,plant administration,and the plant's maintenance program.
Task 4: Write the CPE Report
WQCD's contractor analyzed the information gathered during the CPEs to identify PLFs. A PLF is any
direct or indirect aspect of plant operation that can negatively affect the performance of a SWTP. WQCD
assigned a PLF rating based on the severity of the adverse effect categorized as follows:
Level A: Has major effect on drinking water quality on a long-term,repetitive basis.
Example: inadequate filtration or disinfection
Level B: Has moderate effect on drinking water quality on a routine basis or major effect
on a periodic basis. Example: insufficient plant process control testing
Level C: Has minor effect on drinking water quality on a routine basis or major effect on a
periodic basis. Example: lack of laboratory space requiring samples to be taken
off-site for analysis
Final Report 5-3 CPE Compass Project
;Fo;riid9WILk
Jac a'srn Hrpn ' PhIllias of
t I'eld
I
L. 6. Marian
� Cross Boulder
_. _ _— -
4 6la;kco - vV.shingtor Yuma
_� fir- l HT:-yr r slack.
-_ I Osrfied \-. -° - r snip:gae
I / rants, -
i r a u:tt., ,pr Ca f _Iii
�ir.<ir., t 3 Park 'T� ,, las - _._C nc r to j Kit Carson
la --/ _.._lI k--__
I _meat r
Mesa L„, �, 1 Tcll^ri 6 Paso —jjcr ak noose � k _vu
iA J Ciao Inv
4 lac/ k H F
s.,,na<t.e o-�k
SR n M go!! k sdale _y \
s l\ Bert Provers
_. ��r_1g'l - � " Otero
Dolores Y t s isan -- _._� u
^I r-JI l I i HE ;al J, /
�L acct
II r At)riosa
S
Monte. sr-la -�� _acz I3;All 2 l a P!atit
V k stil'a k---- Baca
Arcnule to Csnajm
- L
0 25 50 100 150 200
Miles
Legend
N
a Surface Water Treatment Plant Location
Interstate Highway FIGURE 5-2
1 County Boundary
SURFACE WATER TREATMENT PLANT
COMPASS PROJECT
Task 5: Send CPE Reports to SWTPs
WQCD prepared reports for each CPE that summarized fmdings and advised SWTPs to address the level
A and B PLFs. Twenty-six of the 28 facilities included in the CPE Compass project had at least one level
A or B PLF. The CPE reports identified PLFs and assisted the SWTPs in improving water treatment
performance generally without requiring major capital expenditures.
Task 6: Make CPE Follow-up Telephone Calls
In 2003,WQCD telephoned each of the SWTPs that received a CPE to determine how they addressed the
level A and B PLFs. WQCD recorded the responses in a spreadsheet and evaluated whether the SWTP's
actions to address the PLF were adequate to eliminate the PLF as a concern. In some cases,WQCD could
not determine during the telephone call whether the action taken by the SWTP was adequate to eliminate
the PLF. WQCD will inspect each of the SWTPs and use the inspection results and MPA tests to
determine if the SWTPs actions were effective in mitigating the PLF(Task 7).
Task 7: Conduct CPE Follow-up Site Visits
WQCD will visit each of the 28 SWTPs to verify the information recorded during the telephone
interviews when it performs a iri-annual sanitary survey at these plants. Also during the tri-annual
survey,WQCD will evaluate the effectiveness of the SWTPs' actions on mitigating the PLFs.
5.3 SUMMARY OF FINDINGS
WQCD quantitatively analyzed the CPE data and the results are sununarized in this section.9 The data are
summarized in Table 5-1 and Figures 5-3 through 5-9 visually represent the data in greater detail.
• Summary of Results
This information is based on telephone interviews and an independent on-site verification has not
yet been conducted. Twenty-six out of 28 SWTPs had level A or B PLFs. Table 5-1 summarizes
the CPE data with regard to the number of PLFs identified,the range of level A and B PLFs,and
the percent of PLFs addressed subsequent to WQCD's site visits.
9 Because the CPE Compass project did not include baseline data to compare to follow-up data or a control group to
compare to treatment groups,no statistical analysis was possible for the CPE Compass project. Consequently,
there is no data analysis procedure or results section for the CPE Compass project and all results are
summarized in this section.
Final Report 5-5 CPE Compass Project
TABLE S-1
PLF RESULTS SUMMARY FOR SWTPs
Range of PLFs PLFs Identified Range Percent
PLF Group (All CPEa) Identified Action Taken
(Each CPE)
Level A and B PLFs 116 2 to 8 55
Level A 48 0 to 3 63
Level B 68 0 to 5 50
• Level A and B PLF Analysis
Overall,there was an average of about two level A and three level B PLF's per facility. Figure 5-
3 presents the total number of PLF's for each facility,distinguishing between level A and B
PLF's.10
• Comparison of A and B level PLF Occurrences
As shown in Figure 5-4,there were 30 percent more level B PLFs than level A PLF's,totaling 68
and 48,respectively.
• Level A and B"Action and No Action Taken"Summary
As summarized in Figure 5-5,there were 48 occurrences of level A PLFs,30 of which were
addressed before the follow-up assessment,and 68 occurrences of level B PLFs,34 of which
were addressed before the follow-up assessment. Figures 5-6 and 5-7 show the frequency of
action and no action taken by number of occurrences for level A and B PLF's,respectively.
10 All figures located at the end of this section.
Final Report 5-6 CPE Compass Project
FIGURE 5-3
NUMBER OF LEVEL A AND B PLFs PER SWTP
J •Number of Level A Performance Limiting Factors
d
s _ l ❑Number of Level B Performance Limiting Factors
6
7
8—
I.,
11
12
13—
Z
E 15
16
18
20
22 a
23
25
26
1 I I
0 1 2 3 0 5 8 7 8
Number of PLFs
Final Report 5-7 CPE Compass Project
FIGURE 5-4
COMPARISON OF LEVEL A AND B PLF OCCURRENCES
ti
1
•
L..aweu. ...Sens. TW W,F.
FIGURE 5-5
COMPARISON LEVEL A AND B PLF"ACTION AND NO ACTION TAKEN"RATES
Level A Level B
18
34 34
30
■Action No Action
rte.
Final Report 5-8 CPE Compass Project
) ) )
FIGURE 5-6
LEVEL A"ACTION AND NO ACTION TAKEN"SUMMARY
Number of PLFs
0 1 2 3 4
2
3
6_ No Level APLFs
9
No Action
15
5 ■Action
10"
12"
22
7
19
I 21
z 20
Q. 26
3 25"
4' _..
17
16
18-
8
14
13" _
11
24-
Final Report 5-9 CPE Compass Project
FIGURE 5-7
LEVEL B"ACTION AND NO ACTION TAKEN"SUMMARY
Number of PLFs
0 1 2 3 4 5
11 No Level B PLFs
d D No Action ICI
20 ■Action
3
6
12
19
25
7
13
4 21
a 9
5
1618
..
I
10
17
15
2
23 I.
24
22
26
Final Report 5-10 CPE Compass Project
Hello