Employment Development Department Strike
Team Detailed Assessment and Recommendations
Prepared by EDD Strike Team Co-Chairs
Secretary Yolanda Richardson, Government Operations Agency
and
Jennifer Pahlka, founder of Code for America and
former US Deputy Chief Technology Officer
September 16, 2020
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p1
Table of Contents
Table of Contents 2
Background 5
Executive Summary 5
Key Issues and Challenges 8
EDD is routing more claims for manual processing than it has capacity to
process. 8
Current Identity Verification Processes Intended to Block Fraud are
Slowing Service Delivery Without Catching Fraud 9
EDD culture must shift from reactivity to rationality as it relates to fraud
11
Organized fraud represents a serious risk to the state, and EDD must
develop capabilities to understand and combat it 13
Staffing realignment must resource the bottlenecks in the system first,
and free experienced claims processors to process claims. 14
The call centers are overwhelmed 16
Call Center 1.0 (8a-12p) 17
Call center 1.5 (8a-8p) 18
Existing Strengths and Innovations 21
The Macro team 21
Expansion of Timeframe to Certify Claims 22
Retroactive Recertification Tool 22
Work Share Application 22
Document Upload 23
Technology Obstacles 25
Current state of technology 25
1980s 25
1990s-2000s 26
2000s-2010s 27
2012-present 28
Claimants getting stuck between BPO and UIO (the EDDCAN issue) 29
CUBS / SCDB Sync Performance 30
System Monitoring 31
Employees are managing passwords insecurely. 32
Some mailing addresses are too long to fit in the database. 32
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p2
Scans of submitted identity verification documents do not include the
color red. 33
Business Process Obstacles 34
No systematic, data-driven way to define and work down the backlog. 34
EDD is not consistently using its own centralized Document Management
(Mail, Scanning) Services 35
The backlog of recomputation tasks is growing. 37
Claims processors are unable to return work items that they cannot
resolve back into the pool. 37
Claims processors cannot always access the necessary skills/queues to
resolve a claimant’s issue in one session. 38
People have multiple open claims at once, and they should not. 39
Out of state wage verification is manual. 40
Claims from newly-separated Servicemembers (Veterans) cannot be filed
online, only on paper. 41
Federal civilian employee claims cannot be filed online, only on paper. 42
Stale work items, some as old as 2013, obscure current claims that require
work. 42
New employees should be reassigned to skills-appropriate work until the
backlog is vastly reduced. 43
The EDD Legislative Office does not have a tool to track legislative
requests. 43
Order of payments on CUBS screen is the opposite of Work Share Form DE
4581WS 44
Claims Representatives Are Not Allowed to Email Claimants 45
Gaps in End User Experience 46
Claimants do not understand their status 46
Claimants do not understand the form questions and make mistakes 47
Claimants receive “zero dollar award” letters they do not understand. 50
Claimants must mail in their documentation. 51
The EDD UIO website does not work on mobile phones. 51
A striking number of people are recertifying on paper and by phone. 52
The usability team is last instead of first in the development process. 53
The EDD UIO website is not accessible to those with disabilities. 53
Individuals who are not fluent in English face insurmountable barriers to
receiving assistance. 54
Users are regularly providing the wrong identity documents. 55
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p3
Claimants are waiting on EDD for information that only Bank of America
can give them. 56
Errors in SSN data entry may be common. 56
The security questions to create an account are unnecessarily complex. 57
Looking Ahead to the Future 58
Redirect the BSM staff back to their previous positions. Reboot the BSM
project. 58
Monitor reasons for disqualified claims. 59
Monitor appeals outcomes for signs of areas to be addressed. 59
Clarify EDD’s purpose and incentives 60
Address the coming spike in holiday determinations workload before it’s
too late 61
Plan to respond to incorrect 1099Gs in January 2021. 62
Simplify policy and automate key processes to enable scaling up and
down to meet future demand 62
EDD employees consider many measures temporary. 63
Conclusion 63
Appendices 65
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p4
Background
Since the COVID-19 pandemic closed many businesses, unemployment insurance (UI) systems
across the country are coping with unprecedented load. California’s UI system, run by the
Employment Development Department (EDD), is no exception. On July 29, Governor Gavin
Newsom directed Yolanda Richardson, Secretary of Government Operations, and Jennifer
Pahlka, former US Deputy Chief Technology Officer, to lead a strike team to set a path for
needed reforms at the Employment Development Department.
The co-chairs of the Strike Team convened a team of experts with experience solving complex
service delivery, operations, and technology problems in government organizations including
HealthCare.gov and the Department of Veterans Affairs, as well as leaders from the California
Department of Technology and the Office of Digital Innovation. This team conducted six weeks
of interviews of staff at all levels of EDD, exhaustive technical systems discovery, and on-site
observation of claims processing. We worked closely with EDD throughout the six weeks,
sharing our recommendations and supporting their implementation along the way. The
following report summarizes our findings and recommendations, some of which have been
acted upon, some of which are in progress, and some of which should be implemented when the
top priorities are complete.
Executive Summary
Ultimately, UI claims processing is a logistics and workflow problem. Every action (nightly
system batches, manual review, waiting for claimant information) runs at a fixed speed (time
per item), with a relatively fixed throughput (total items per time). In some cases steps can be
sped up, or throughput increased, by buying faster computers, or increasing staffing. A concrete
example of throughput from our field data: an experienced claims processor can manually work
through only 24 claims per day.
Claims may be worked on, queued for work, or stored by many different systems or staff,
depending on a large number of variables. Variables in the claim application itself such as name,
date of birth, tax identifier, employer are among these. There are also variables in data sourced
from other locations, such as the Social Security Administration, Franchise Tax Board, or
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p5
Department of Motor Vehicles. Lastly, there are variables related to the claims process itself,
such as manual actions taken by EDD staff or contractors.
A detailed breakdown and map of this process, to the extent made possible by the data we've
gathered, is included later in this report. It is important to understand that there are effectively
many paths a claim can take during processing, which all have different phases, each with its
own speed and throughput. The lowest speed, lowest throughput phases are all the ones which
depend on EDD staff taking action, or "manual" phases.
As long as the inbound rate of claims per day exceeds what the slowest, lowest throughput
part of the system can complete in a day, the backlog of undetermined claims will grow
unbounded until the new claims rate decreases.
Based on data of new claim filings, California’s current backlog of undetermined claims is
growing by at least ten thousand claims per day. Given the state of the economy and the
likelihood of future legislation, executive orders, or other policy to support workers, it's
reasonable to assume inbound claim rates will not abate in the foreseeable future.
EDD's maximum throughput for manual processing of claims never exceeded 2,400 per day in
2019 or 2020. Recent new UI claim reports show that right now, more than 20,000 claims per
day are diverted into manual processing, mostly for identity verification.
In the following report, the Strike Team presents a detailed picture of where claims can move
quickly, where they do not, and for what reasons claims get diverted to slower,
lower-throughput processes.
Our top recommendations follow. More detailed implementation suggestions, including
suggestions of specific measurements, are found elsewhere in this report, and in additional
deliverables, including 10 Appendices.
Our most important recommendations:
● Measure every material step of the claims process on a daily basis. Include tracking of
reason codes and throughput of each phase.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p6
● Quickly procure and deploy a NIST IAL2-compliant Identity Verification solution for all
claimants, integrated as early as possible in the onboarding process.
● Adopt the “burndown” plan EDD has proposed in the memo titled “Governor’s Office
Strike Team Recommendation to delay new claims filed and redirect highest skilled staff
to focus on processing claims,” based on the model provided by the Strike Team, and
refined and optimized by the EDD team. In this analysis, older claims will begin to be
addressed immediately, and with relief coming to many claimants as soon as a few
weeks from now, and the final remnants of the backlog resolved within four months.
This plan reflects incredibly disciplined, thorough, and thoughtful work on the part of
the EDD team and is by far our best chance of getting claimants the relief they need. All
stakeholders should vigorously support the staffing changes in this plan.
● Complete the deployment of the current document upload, as a long-term solution for
all types of documents, and make it accessible to more claimants by making it
mobile-friendly.
We are pleased to report that EDD has embraced each of these recommendations and is now
leading the department towards their implementation, pending approval from the Governor on
the burndown plan. CDT and ODI continue to collaborate effectively with EDD on the technical
pieces of the implementation and feel confident in their success.
Finally, the only way out of this unbounded growth is ongoing measurement and action. EDD
must continually analyze data showing the speed and throughput of the systems & processes at
a resolution sufficient to find low-throughput bottlenecks. That data can then be used to make
high-confidence decisions about process, people, and technology in order to increase
throughput. The Strike Team has engaged with EDD in this kind of analysis and
decision-making throughout the 45 days of our assignment, and we are excited to see EDD
continue this practice far into the future.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p7
Key Issues and Challenges
EDD is routing more claims for manual processing than it has capacity to
process.
EDD received an average of 60,368 new electronic claims each day during the week ending
August 22, 2020. Of those, an average of 40% were routed for manual processing: 24,147
manual claims per day. Once you require manual processing, you cannot have your claim
determined in under 21 days.
From November 2019 through January 2020, by all accounts, EDD processed claims in a
reasonable timeframe. EDD averaged 2,312 manual claims per day during this time.
Today’s claims are also generating more work items per claim:
The breakdown of reasons an electronic claim is sent for manual review is known and is
available in an internal report called CPT Conditions.
If EDD can automate 94% of claims (from a current baseline today of ~60%), it will reduce
the number of claims requiring manual attention to 3,622 per day, a number which it has
capacity to complete. Moving to an automated identity verification provider would handle an
estimated 77.69% of these manual claims, increasing the automation rate to 91%.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p8
The reasons that a claim is rejected from the automatic process and sent for manual review are
serial; it’s possible that a claim has multiple issues requiring manual review, but only the first
reason encountered is captured. Still, this is an excellent baseline for where to focus more
automation efforts.
Recommendations:
● Implement the plan outlined in Appendix A: Data-Driven Definition & Resolution of the
Backlog and review these numbers weekly, design improvements to address the largest
causes of manual review, test how well these improvements are working, and continue
addressing root causes on a weekly basis.
UPDATE: This has begun and should continue.
● Implement the plan outlined in Appendix B: Acceleration Plan for Identity Verification to
verify claimants’ identity automatically, using federal NIST standards for identity
verification, and completely replace its current identity verification system and process
immediately (no later than September 30, 2020).
UPDATE: This plan is in progress. The vendor selected is ID.me.
Current Identity Verification Processes Intended to Block Fraud are Slowing
Service Delivery Without Catching Fraud
The U.S. Department of Labor requires that EDD not exceed a 10% improper payment rate
(which they have done - most recently it was 8%); in order to do that, it must prevent
fraudulent claims from going through the system. In nearly every interview, leadership and
field workers alike emphasized that a large part of their job was identifying “fraudsters” who
are impersonating other people to collect benefits. One of the ways EDD tries to do this is by
flagging claims for further identity verification. Currently, identify verification is the
overwhelming driver (78%) of claims being sent for manual processing.
Organized high-volume unemployment fraud is a very real problem in every state, and a
frequent topic with the National Association of State Workforce Agencies (NASWA). However,
the current automated identity verification process is simplistic and easy to exploit. Anyone
attempting to intentionally claim on behalf of someone else fraudulently can submit an
application that goes through the automated process, especially if they have the person’s SSN.
During the course of this engagement, the Strike Team heard frequently about the fraud
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p9
reported recently in the media (of inmates receiving unemployment, or of one woman getting
50 letters addressed to her house). In these cases, the claimants are going through the automated
process; the manual process did not stop this fraud. EDD’s practice of manually processing a
high number claims for further identity validation appears to catch relatively few cases of fraud.
The Strike Team reviewed data from the Identity Verification Database to explore this. Of
183,167 cases in the three months of May, June, and July this year, less than one half of one
percent (.04% or 804 people) were deemed imposters during EDD review. But even that small
fraction may still overstate the imposter rate. Of appeals for identity verification (Issue
1257(a)), from January through June 2020
1
, the California Unemployment Insurance Appeals
Board overturned EDD’s decision 53% of the time (9,519 out of 18,111 appeals). That would
suggest that about half of that .04% were in fact legitimate claims, leaving .02% of those claims
as likely fraud.
But the cost of finding that small number of imposters is extremely high, both in terms of the
literal cost of staff time and in terms of the negative impact on the system. In those same three
months of May, June, and July this year, EDD routed 1,383,302 claims through manual
processing, primarily to further validate their identity, in the hopes of flagging what appears to
be .02% of claimants who can be identified as imposters. The impact on those 1,383,302
claimants is significant. Because EDD lacks the capacity to process this volume of manual claims
and to answer calls from confused and worried claimants, many of them are stuck in limbo with
little visibility into the status of their claim. It is the view of the Strike Team that the minimal
fraud prevention derived from these practices does not justify the high cost to the system, to the
overall pool of claimants, and to the trust and faith of the public in the ability of the system to
serve them.
During the course of the engagement, the Strike Team and EDD have partnered to very quickly
select and begin the implementation of an automated solution that meets a far higher standard
of identity verification, as mentioned above, and we believe this will greatly reduce fraud
moving forward. We commend EDD for their championing of this effort and their speed to act,
in partnership with CDT and ODI.
Recommendations:
1
https://cuiab.ca.gov/Statistics/quickStats/Outcomes_Jan-Jun_2020.pdf
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p10
● (As above) EDD should follow our proposed plan in Appendix B: Acceleration Plan for
Identity Verification to verify claimants’ identity automatically, using federal NIST
standards for identity verification, and completely replace its current identity
verification system and process immediately (no later than September 30, 2020).
UPDATE: This has begun and should continue.
● EDD must make a plan with IT to automatically reconcile situations where a new UI
claimant’s SSN is already in SCDB (e.g. for a past DI claim) before this new identity
verification system goes live. We have recommended a new, more nuanced rule for
adoption for the short-term, but in the long-term, this conflict needs a permanent
resolution.
● Until there is no backlog of correspondence (including electronic messages and physical
mail), claims should not be assumed to be fraudulent or abandoned by the claimant. See
recommendations in the section on Business Process Obstacles for specifics on
centralizing mail and document processing.
EDD culture must shift from reactivity to rationality as it relates to
fraud
In our interviews and observations, stories and anecdotes about fraud and/or suspected fraud
were often used to explain why EDD could not act quickly to avoid the growth of the backlog. For
example, the Strike Team suggested that EDD turn off certain conditions that unnecessarily flag
claims for manual processing until Id.me was operational, based on the analysis described
above. (These flags included conditions like the applicant including a middle initial in her UI
claim, but the SSA database showing no middle name, or vice versa.) In EDD’s view, it would
have been inappropriate to reduce fraud prevention measures at a time when the sheer volume
of applications suggested increased fraud; in the Strike Team’s view, the tradeoff was worth it,
as those flags were not functioning as fraud prevention, but were preventing the timely
payment of hundreds of thousands of legitimate claims. EDD delayed acting. If those conditions
had been turned off when the Strike Team recommended them, EDD could have avoided
sending hundreds of thousands of claims to manual processing during the spikes in new claims
that occured at the end of August.
There has developed at EDD a culture of allowing fear for fraud to trump all other
considerations, and a lack of analysis of fraud avoidance tactics to understand both their
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p11
effectiveness and their effect on the core mission. This was visible in several areas. For example,
EDD staff and leadership told the Strike Team that the tens of thousands of people who have not
responded to identity verification requests are fraudulent. This is possible, but with current
practices, it is impossible to know, and there is ample evidence of other explanations. There are
currently 7,695 pending work items for addresses that are “too long” to fit in the database, and
therefore any mail sent to those addresses would be coming back as undeliverable. We heard
from every legislative office we spoke to that their constituents had mailed in the requested
documents to validate their claim, often multiple times, but received no acknowledgement that
EDD had received them, and remained stuck in limbo. Our observations of unopened mail
during visits to the field offices lends credence to this. Until it is processed and quantified,
returned mail should not be assumed to indicate fraud. As discussed in the section on Gaps in
End User Experience, individuals who do not speak fluent English have almost no assistance to
make it through the process, and even fluent English speakers frequently complain that they
don’t understand EDD notices. We encourage EDD to investigate further the possibility that a
lack of response from some claimants may indicate confused and desperate people who may
simply have given up, and to use data-driven methods to make this determination.
While it is certainly EDD’s job to fight fraud, it is also EDD’s job not to allow the fight against
fraud to interfere with the delivery of benefits to legitimate claimants. We believe there is work
to be done by EDD leadership to shift the culture of the organization from reactive to
data-driven in this area. This will help the department deliver much-needed services to
Californians in need while also being good stewards of public funds.
Recommendations:
● EDD leadership should set clear expectations with all staff at all levels that fraud
prevention and detection practices must be supported by data and evidence and that
justifications for new and existing anti-fraud practices include an analysis not only of
their effectiveness, but also tradeoffs and unintended consequences of these practices,
including adverse impact on the experience of all claimants. EDD should seek support in
actively changing the culture of the organization, but the work they are doing now to
address the backlog with a data-driven plan will serve as an excellent platform for this
culture shift, especially because it has such strong leadership support.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p12
Organized fraud represents a serious risk to the state, and EDD must develop
capabilities to understand and combat it
The Strike Team was not charged with analyzing fraud; our mandate was to recommend ways to
improve the claimant experience, and we largely limited our investigation to the connections
between ineffective fraud-fighting practices and EDD’s productivity and throughput, in order
to speed claims processing.
However, though practices intended to reduce fraud have been the key driver of the large
backlog of unprocessed claims, and we note a reactivity to fraud and its negative implications to
service delivery, there is evidence that California may in fact be the victim of significant
organized fraud. The state is not alone in this; many states report sophisticated, organized
fraud attacks and struggle to respond effectively. In a recent news article, Arizona stated that
out of 1M PUA claims, almost 900,000 were deemed to be fraudulent
2
. It is widely acknowledged
that this is not the result of 900,000 Arizonans attempting to individually file fraudulent
claims; rather it is assumed to be organized crime. Other states believe that they are the victims
of attack, but don’t know how many cases there may be.
About four weeks into our six-week effort, EDD leadership asked for help in responding to a
significant spike in PUA cases, up to almost 120,000 a day on August 31 and September 1. CDT
stepped in and secured the help of the California Cybersecurity Integration Center (CalCSIC) and
their forensic specialists.
Recommendations:
● As stated in the section above, EDD should follow our proposed plan in Appendix B:
Acceleration Plan for Identity Verification to process identity verification automatically in
order to reduce certain kinds of fraud.
● Once the new identity verification solution is in place and the process meets federal
NIST standards, EDD should continue working with CalCSIC to thoroughly investigate
potential fraud and explore the offerings of vendors who have state of the art technology
that might be a fit for California’s needs. EDD should further reevaluate how they
2
https://www.azcentral.com/story/money/business/consumers/2020/08/20/arizona-departme
nt-economic-security-delayed-payouts-pandemic-employment-assistance-applicants/56171
65002/
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p13
identify and manage the risk of fraud given that the vectors for fraud under the new
system will be different.
● As stated above, EDD leadership should set clear expectations with all staff at all levels
that fraud prevention and detection practices must be supported by data and evidence,
and model that behavior for staff.
● EDD’s fraud team has been until now charged with supporting a set of practices that will
be irrelevant when ID.me is operational. The resources the department dedicates to
fraud need to shift towards different skill sets, including forensic analysis and cyber
security coordination.
Staffing realignment must resource the bottlenecks in the system first, and free
experienced claims processors to process claims.
Since January 2020, the number of staff deployed to unemployment-related tasks has grown
exponentially, fueled by reassigning existing EDD employees and by fast-tracked hiring.
Office
January 2020
August 2020*
December 2020
(goal)
ID verification
20
460 (288)
503
Recomputations
42
557 (335)
757
Claim filing
546
933 (143)
1,966
Work Share
51
78 (27)
251
*Number in parentheses is the number of redirected EDD staff reflected within the August total, as opposed to brand-new hires.
By end of year, the goal is to return all redirected staff and replace them with all brand-new hires.
EDD should be commended for moving quickly in the face of a crisis; however, these actions
have had unintended consequences, and the plan needs realignment. Processing unemployment
claims is specialized work, requiring an 800-page training manual; processing them across
multiple fragmented, bespoke IT systems in the context of complex rules and regulations is
even more specialized. Many kinds of work cannot be completed by new employees at all. In
recomputations, for example, it takes three years of experience, schooling, and formal testing
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p14
to progress from an entry-level Accounting Tech to an Accountant I, and an additional three
years to progress to each subsequent tier. Across multiple days, we observed zero work items in
the Accounting Tech queue, and BCIS queue data confirms this. There is no work for the 515 new
recomputation employees to complete.
Current conditions have made it even harder to effectively add new staff. The addition of new
programs brings an increase in complexity to an already complex system. The pandemic forced
an abrupt move to remote training, which is difficult to get right under the best of conditions,
and nearly impossible when training cohorts have 180 participants, as happened at EDD, and
supervisors have at least 50 employees. Remote work is even challenging for experienced
employees; Employment Programs Representatives (EPRs) solve complex problems, and are
used to collaborating across cubicles to solve claimant issues. EDD has made significant efforts
to reduce the training time for new employees, but it remains the case that gaining the skills to
process claims takes six months.
When the Strike Team began our work, experienced employees had been redirected to training
new employees, and were not able to focus on working claims. Since the new employees cannot
process claims, and will not be able to process claims anytime soon, productivity for key tasks
had gone down despite adding staff. We analyzed the DPT (Daily Production Tally sheet) data
from January 2020 and from July 2020, measuring individual employee productivity. It now
takes 2-5 times as long to do recomputations, for example, even with redirected EDD staff who
are already familiar with the computer systems.
While an influx of new staff reduced productivity in some areas, they can increase throughput
in other areas. As we stated earlier in this report, as long as the inbound rate of claims per day
exceeds what the slowest, lowest throughput part of the system can complete in a day, the
backlog of undetermined claims will grow unbounded until the new claims rate decreases. The
slowest, lowest throughput parts of the system include identity verification, the CPT queues
(routing claims requiring processing to the correct next step), and mail sorting, processing, and
tracking. Today, these areas will benefit from additional staff, until those queues are cleared, at
which time further reassignment will be needed.
The Strike Team has provided EDD with a model for understanding where changing staffing
levels in particular areas can increase system throughput. EDD has embraced this model and
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p15
refined it over the course of the final week of the Strike Team’s engagement, and the result is a
plan to address each and every claim in the backlog with the maximum possible speed and
efficiency. In this analysis, older claims will begin to be addressed immediately, and with relief
coming to many claimants as soon as a few weeks from now, and the final remnants of the
backlog resolved within four months. The plan reflects incredibly disciplined, thorough, and
thoughtful work on the part of the EDD team and is by far our best chance of getting claimants
the relief they need.
Moreover, EDD’s comprehensive, data-driven approach to this plan should be commended,
along with the leaders within the department who championed it. It should inspire confidence
from all stakeholders, as it is emblematic of an approach that, if applied consistently to future
challenges, will mean a dramatically different and better future for the department.
Recommendations:
● Adopt the “burndown” plan EDD has proposed (which models where staffing is needed)
and support the associated staffing changes.
● Do not allow new hires to work overtime hours for the first three months, since those
hours only create additional work for experienced employees.
● Allow all redirected employees to return to their original desks to the greatest degree
possible.. We found multiple employees who had been temporarily reassigned to
different floors/areas, which disrupts their ability to collaborate with their original
teams.
● (As above) BCIS should build the functionality to allow employees to return work items,
without requiring a supervisor to move them, and without requiring that the work item
be reassigned to another employee.
The call centers are overwhelmed
EDD has two call centers: one is open from 8am-12pm, and is staffed by experienced
Employment Programs Representatives, (EPRs) with the ability to make changes to claims.
This is called “call center 1.0” in some documents. A second call center, variously named
“Technical Assistance Line” or “call center 1.5,” is open from 8am-8pm, and is staffed by
hundreds of new employees for the purposes of handling lower-level requests.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p16
Call Center 1.0 (8a-12p)
The phone number 1-800-300-5616, now called “call center 1.0,” was not intentionally
established as a call center; it was a convenience number connected to the desk phones at a
particular Northern California field office. This phone number can’t be routed anywhere outside
of the in-office desk phone system, so as employees began telework and/or were reassigned to
other priorities (e.g. Work Share), they stopped answering the phones.
This line received on average 6.7 million calls per week during the month of July, representing
approximately 600,000 unique callers. The outbound call message also refers callers to the call
center 1.5, which is discussed below.
There is confusion among employees and management about this call center. When we visited
the office where it is housed, and asked where the call center was, an employee speculated that
perhaps it had moved to southern California, but that was determined to be inaccurate. A
manager explained that because most of the staff is now working from home, and those that
still come in have been redirected to other priorities, this office no longer answers a high
volume of calls. This redirection is corroborated by the workforce-monitoring tools that EPMs
use, including CCPulse and VCC.
Prior to the pandemic, in February, the enterprise-wide average for this call center was that one
employee could handle 5 calls per hour. (EDD’s current public estimates are that agents can
only handle 4 calls per hour.) EDD would need 6,000 agents working from 8-12, without
competing priorities, to serve 600,000 unique callers at 5 calls per hour. Observing the
supervisor dashboard “CCPulse,” we found no more than 20 employees answering calls at a
time, and the number of empty cubicles we saw would support that number. We can safely
estimate that no more than 1 in 1000 people that are trying to reach this call center on a given
day are getting through. The morning our team visited this office, we observed four live phone
calls in 90 minutes. Of those, two were determined by staff to be from claimants attempting
fraud, and they were not helped. On each of the calls, much time was absorbed hearing the
claimant explain the number of hours they have been on hold, and the number of days they
have been trying to call, highlighting how challenging it can be to serve claimants efficiently
under current circumstances.
In summary, in “call center 1.0”:
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p17
● Capacity is down more than 80% from the pre-pandemic level.
● Demand (incoming call volume) is up over 1000% from the pre-pandemic level.
● Some callers are not being helped because of fear of fraud.
● Callers who don’t get through are referred via the outgoing message to the call center
1.5, which we discuss below. Because that call center can’t help them with claims, many
callers then return to trying to get through to call center 1.0.
The net effect is that 600,000 unique callers a month are waiting on hold for hours without a
statistically significant chance of being served, just in call center 1.0.
Call center 1.5 (8a-8p)
The Technical Assistance Line, also known as call center 1.5, will eventually have two levels:
“Level 1” password reset requests and “Level 2” higher-level requests handled by
journey-level claims representatives. Currently, however, it only has “Level 1” employees.
Newly hired claims processors are assigned to this call center, but as new hires, they are not
able to move claims forward and are effectively at the same level as the call center agent. The
platform for call center 2.0 is the Verizon product “Virtual Contact Center,” or VCC.
In July 2020, the EDD received an average of 3.5 million calls per week through
the Technical Assistance Line, representing on average 412,000 unique callers. Many of these
are referred from call center 1.0’s outbound message. As of August 4th , EDD has 1,561 agents on
the technical assistance line.
This is EDD’s hiring plan for the call center:
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p18
By the end of October 2020, EDD plans to merge the employees from call centers 1.0 and 1.5, and
have one call line, referred to as call center 2.0.
EDD has been reporting on measures of number of calls answered. This measure has been
improving as more staff are hired. However, in order to serve claimants and reduce the backlog,
the measure that matters is how many calls are resolved. Using “Voice of the Customer” data
provided by Verizon to EDD on a weekly basis, the team calculated that the highest-possible
first-call resolution rate for the first week of August, when the 8am-8pm call center answered
167,443 calls, was 1.4%. This almost exclusively represents people calling for password resets,
which is the only statistically-significant call reason that these new employees can handle. The
vast majority of calls are for things the call center cannot do, like providing claim status or
fixing applications. Hiring more staff that cannot handle the actual reasons people are calling
will improve the calls answered statistic but will have no impact on resolving issues with claims.
It is worth noting that the federal government does not provide funding for the call centers; this
cost for this call center falls fully to California taxpayers.
It is also worth noting that many other recommendations in this report, some of which are
underway, including the implementation of ID.me and stopping the “zero award dollar notices”
should have the effect of reducing the number of calls. In addition, the eventual launch of the
claims status tracker, also discussed below, will also help serve claimants without needing to
call.
Desperate claimants are finding and sharing other EDD numbers that are not meant for the
public. The TTY phone line (for hearing- or speech-impaired claimants) used to receive four
calls a month; it now gets 40 per day. When we observed calls in the Work Share office, none
were not related to Work Share, but were rather traditional claimants with questions about their
claim. If this pattern continues, every employee with a telephone will be overwhelmed with calls
they are unlikely to be able to answer, and that will prevent them from working the claims in
front of them.
Recommendations:
● Shift call center 1.0 to an outbound call strategy, in which experienced claims processors
actively call claimants if they need to speak to them to resolve their claim, for until the
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p19
backlog is significantly reduced. It is critical to keep experienced claims processors
focused on the tasks only they can perform. Colorado has taken this approach, so there is
precedent.
● Call center 1.5 will need to remain open, but must be retooled to set appropriate
expectations with callers so that fewer claimants spend less time needless on hold, and
so that the activity of this call center stops reducing the productivity of the experienced
claims processors. Outbound messages must be carefully crafted to quickly direct callers
whose needs can be resolved elsewhere (such as Bank of America).
● Reassign new call center employees to opening, sorting, and processing inbound mail
and other tasks that help reduce the backlog. Stop hiring any new ones.
● Continue the transition to VCC so experienced claims representatives can make
outbound calls while working remotely.
● Short of this, update call center welcome messages so the top 10 reasons (below) for
callers are represented in the hierarchy of numbers.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p20
Existing Strengths and Innovations
The Macro team
EDD has a very valuable asset in its “macro team” and “macro server farm.” This is the largest
reservoir of adaptive capacity in EDD’s technology universe, meaning that it is the best place to
start when trying to figure out solutions to new demands and unexpected problems. Several
people, from line workers to executives, cited “the macro guys” as their main hope for getting
through the implementation of the new federal programs. They do not point to the “mainframe
people” or the “UIO people,” showing that they implicitly understand these other systems will
be slower to adapt. The "macro team" is indispensable, and they are critical to all technical
processes and changes in claim processing.
It is important that their priorities always be the same top three priorities of the department.
Eric Weber’s macro libraries exist today as a de facto mainframe API, and one that is much more
functional that we have seen in other agencies that have dedicated two years to the task.
Although there are limitations to what you can accomplish through macro automation
(described above), we also heard that the macro server farm has had to turn down or postpone
work for lack of hardware.
Recommendations:
We are recommending that EDD immediately expand this capacity as much as is practicable.
We’d encourage EDD to do it in the simplest possible way, which is to just add more physical
workstations, provisioned in the same way on the same subnet as the existing macro server
farm. There are fancier ways to do this, but it won’t be possible to make big changes without
losing its current virtue of being simple, well understood, and operational today. If EDD
attempted to upgrade or migrate the current server farm, it could lead to loss of this critical
capacity.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p21
Retroactive Recertification Tool
During the launch of Pandemic Unemployment Assistance, EDD temporarily paused weekly
certifications to reduce expected load on the system and simplify the claimant experience. This
was an excellent attempt to speed the claimant experience. Unfortunately, the Federal DOL
disagreed with this interpretation and issued guidance requiring regular recertification for PUA.
EDD worked with Nava PBC (a vendor) to rapidly deploy a retroactive certification tool. This tool
allows a positive user experience for claimants while meeting an unexpected rule requirement
from DOL. The creative and functional mechanism for tying data back to the database, and focus
on user experience should be emulated in future improvements.
Work Share Application
Field staff demonstrated the new Work Share online application for us, and are generally very
positive about the application. By early accounts, it will enable greater levels of automation for
the Work Share program, which is currently entirely paper-based -- and which more employers
will be encouraged to use, to at least partially employ more people. It also uses modern
technology (Salesforce) that does not need to be maintained or upgraded by EDD IT, and its
vendor has demonstrated the ability to rapidly iterate and deliver feature improvements on a
weekly basis in collaboration with the workforce.
We have a few suggested recommendations for the continued success of the Work Share
application:
● Make the “holiday schedule” page easier to access. Every employee we shadowed
struggled to get to this page, which they needed in order to complete most DE 4581WS
forms.
● Partner with the WCUG team to further improve usability and plain language for
employers. Staff report that employers continue to be confused by certain fields on the
Work Share forms.
● Add structured “normal hours” for each employee in Work Share plans for easy
comparison.
● Build functionality to automate holiday determinations (see below).
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p22
Document Upload
We applaud EDD for recognizing that electronic document upload is a critical necessary feature,
and for partnering with an industry commercial offering (Docusign) instead of building
something from scratch. These were good decisions, with document upload live as of September
1, 2020 in a limited capacity and expected to expand in the coming weeks.
A concern about this approach is that the document upload does not work on mobile. Currently,
40% of people file their unemployment claim via mobile device despite the fact that UIOnline is
not meant to be viewed on mobile. We would anticipate an even higher percentage would
attempt to use their mobile phone to submit documents, since they would use their phone’s
camera to photograph them. Making the document upload work on mobile should be the
immediate next piece of work, along with making UIO mobile responsive. Those are two
separate, but related, endeavors.
There are outstanding usability issues with document upload, already documented by the
Website Content and Usability Group (WCUG), e.g.:
● Users frequently skip the checkbox confirming if they applied for unemployment. They
are then confused as to why they don’t see the paperclip icon as mentioned in the screen
text. There is no feedback in place to explain why this happens and how to correct it.
● Multiple users did not understand they had successfully uploaded a file. Seeing the file
name was not enough of a cue and requires better feedback. This confusion prevents the
claimant from moving forward and will drive them to call the EDD for help.
There are also outstanding accessibility issues with document upload that are easy to fix that
are documented by the Digital Accessibility Team (DAT) team, e.g.:
● Using tables for formatting, which screen readers have a difficult time parsing. This
issue effectively makes the page unreadable to users with screen readers.
● Missing language attribute, which can cause screen readers to read the page incorrectly.
● Inaccessible color contrast, which can prevent vision-impaired users from being able to
see the page.
● Undefined headings, which prevents screen readers from being able to read the page in
order.
● Incorrectly nested lists, which prevents screen readers from being able to read the page
in order.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p23
● Improper use of bolded content, which influences how screen readers emphasize text on
the page to users.
● Improperly tagged PDF links, which prevents screen readers from reading them
correctly.
● Inaccurate descriptive content to “click on a paperclip” that does not translate to a
screen reader.
The WCUG and DAT teams should be incorporated into the beginning of the development and
release cycle of this mission-critical functionality. It is concerning that they tested,
documented these findings, and even provided a video, and yet the release proceeded without
addressing them. A number of these issues are so simple to fix that there is no excuse for why
they were not addressed prior to release.
Document upload should also be expanded as quickly as possible to all documents, as this would
enable wage verification, military claims, and federal employee claims to file online.
Finally, document upload should be instrumented immediately, in the next release, with Google
Analytics to objectively measure usability and user drop-off rates.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p24
Technology Obstacles
Current state of technology
Like most government agencies, EDD has gotten along for decades by adding new layers of
automation to the systems that already existed. They are built in several contrasting
technology paradigms from different eras. Since each new system depends on everything that
came before, each successive layer tends to become more constrained, less capable, and more
fragile. Meanwhile, since old systems have multiple new systems that depend on them, they
have become difficult and expensive to modify.
1980s
The bedrock technology, and ultimate source of truth and system of record for all claims and
payments, is a DB2 relational database running on an IBM mainframe operated by the CA Office
of Technology. We did not attempt to document its origins. It’s likely it was developed in the
1980s, when this technology dominated new projects. It’s also likely that the Office of
Technology today runs much more recent and capable hardware, since the platform is still
actively sold and supported by IBM.
The Single Client Database or “SCDB” is a user application that runs on the same mainframe
and DB2 database. Its user interface is based on text screens for TN3270 terminals. An EPR
doing their daily work has a TN3270 emulator (which they call “the GREEN SCREEN” in audible
caps lock, or sometimes “Otech” because that appears on the login screen) occupying one of
their two monitors almost always. They tend to use the mainframe interface directly whenever
they can.
Most of EDD’s automated business processes are executed in batches overnight on the
mainframe, with thousands of COBOL scripts computing millions of transactions.
All of the above is actively supported and developed by the IT Branch’s Application Services
Division, which has about 60 full-time employees dedicated to mainframe operations.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p25
1990s-2000s
When mainframe terminals were replaced by Windows desktop PCs running emulators, it
became possible to write scripts to simulate UI actions in the terminal. EDD employees on the
front lines of data entry immediately put this to use, constructing “macros” to automate
repetitive actions. The 3270 terminal emulator as used by EPRs today has several rows of
brightly-colored buttons comprising dozens of such macros.
The next evolution was when one or two successful macro writers transferred to IT Branch and
learned Visual Basic, and then the C# and .Net programming environment, giving them the
ability to automate more complex tasks and run larger batches.
Today, the screen-oriented 3270 interface of SCDB has been abstracted into a sophisticated
XML model interpreted by a .Net library, that is updated in coordination with SCDB releases.
The organization still calls this type of automation “macros.”
A few dozen workstations, known as the “macro server farm,” are dedicated to running one-off
“true ups” and other odd jobs through this middleware.
All of the above are actively supported and developed by a handful of full-time employees which
are shoehorned into the UI Online section of the Product Development Division of ITB.
EDD’s current “macros” are far removed from their origins, which were simple
automations of rote mouse and keyboard movements. But what is still true, and important to
understand, is that “macros” are automations of user actions on the SCDB. Novel functionality
or new data semantics always have to be implemented in SCDB first, and anything that can be
done via a “macro” can (by definition) be done in SCDB directly.
The fact that both layers are alive and in use by EPRs means that for any given task, there are
multiple ways to do it. The knowledge and habits are fragmented differently in each EPR, which
increases the difficulty of training new hires.
The “macro” libraries represent a de facto mainframe/SCDB/DB2 API, and are in fact the only
such API in use by EDD today. This means that, outside of COBOL, macros are the only way that
EDD can automate significant new workloads.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p26
2000s-2010s
An early translation of the paper forms into a “web 1.0” submission form was called eApply. It
survives today only in historical references, such as the names of some work queues and
concepts in the UI Online application that replaced it.
2012-present
An effort was made to translate both the internal and external interfaces into the web
application paradigm. The California Unemployment Benefits System (CUBS) launched in 2013,
recreating many of the claims maintenance tasks in a Web browser window. An Internet
Explorer window open to CUBS occupies the second monitor on an EPR’s desk most of the time.
CUBS runs on Windows and C#, and it depends on both the SCDB and its own Microsoft SQL
Server data store. The SCDB interface has performance issues which we describe later. The SQL
Server database was used to implement “work queues,” which are how EDD represents most of
its manual business processes today.
The basic idea of a “work queue” is that tasks requiring human action are represented by rows
in a “work item” table. Each item has a queue type. Specific groups of workers are granted the
ability to “reserve” work items. When they are resolved, they are removed from the queue.
Resolution often means other work items have been created in other queues. The
implementation of work queues is straightforward enough. As a business process, there are
some obstacles, which we describe in “Business Process Obstacles” below.
131 distinct work queues were defined at launch time in CUBS. They have not all been used as
expected, and 22 have been added, for a total of 153.
3
After CUBS, a similarly structured public-facing interface launched called “UI Online” or
“UIO.”
4
This is where over 80% of new claimants begin their process today.
Also around the same time, a separate system named Benefit Claim Information System (BCIS)
was constructed to manage identity and “recomps.” It is the same technology as CUBS, with an
3
This is based on CreatedDate from the table tblWorkItemType in the UIMOD database.
4
Online at https://edd.ca.gov/Unemployment/UI_Online.htm
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p27
extremely similar interface, and likely shares a lot of code. Nevertheless, it was contracted
separately and uses a separate SQL Server data store.
Having the same internals as CUBS, BCIS also has its own work queues that are not connected to
the ones in CUBS.
All of the above were contracted to Deloitte and are currently maintained by a combination of
Deloitte and EDD staff. EDD has 16 full-time employees in the UI Online section of the Product
Development Division of IT Branch.
Claimants getting stuck between BPO and UIO (the EDDCAN issue)
In order to file an application on UIOnline, you must first create a BPO account. From within
that account, you create a UIOnline account after filing an initial unemployment claim.
However, once your UIOnline account is linked to a BPO account, you cannot link it to any other
BPO account. This means that someone who forgot about a pre-existing BPO account and
accidentally creates a new one, is now “stuck” and cannot get into UIOnline (because their
UIOnline account is attached to their previous BPO account).
Only “clean claims” (fully-automated claims) were auto-registered to UIOnline before the
pandemic. Everyone else could file an initial claim, but must wait to receive an “EDDCAN”
number in the mail to register for the UIOnline account in order to access any additional
UIOnline functionality (like recertifying, messages, etc.) After the pandemic, claims that went
to manual for reasons other than identity verification were auto-registered, so those claimants
did not have to wait for the EDDCAN in the mail. Once automated identity verification goes live,
this issue will be fully resolved.
An additional, though more rare, issue is claimants registering with an ECN (EDD Control
Number) instead of an SSN. This ECN number is only meant for internal and temporary
purposes, and claimants should not be able to register with this number. To date, 1,509 people
have created an account using an ECN (identified as starting with 999 or 990). Though they are
flagged in the system, they must be manually resolved.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p28
In July 2020, 55,555 of 177,527 AskEDD messages (one-third) were related to requesting an
EDDCAN number.
Recommendations:
● This issue will be resolved with the launch of the Id.me solution.
CUBS / SCDB Sync Performance
CUBS is a web-application interface that operates on SCDB (mainframe) data. This means that
when changes are made to claims, those changes have to be written or “synced” to SCDB. EPRs
told us that they regard wait times of 10-20 minutes for the “sync” to be normal.
One long-tenured manager of EPRs told us anecdotes about mysterious “sync” issues such as
completing a large batch of work, checking in the SCDB, finding that none of the updates are
visible, starting the batch over again, and discovering hours later that the updates finally
propagated.
Our investigation could not conclude if these problems are currently affecting users,
apocryphal, or happened in the past and are now solved. Nonetheless, they cause EPRs to treat
CUBS as fragile, and they are afraid to start a new work item or disturb the web browser in any
way while waiting for the “sync.”
Visiting the contact center on two consecutive days, we directly observed two periods of slow
performance where the CUBS sync took several minutes. We followed up with the IT Branch
directly, and they responded a day later to confirm that there had been periods of congestion
which they are hoping will be resolved by an upcoming SAN
5
upgrade. We asked how these
systems are proactively monitored, but did not receive any dashboard screenshots or other
evidence that such monitoring is in place.
We also investigated the contact centers’ channel for reporting and escalating performance
problems. The line workers and managers are all aware that there is an “IT Helpdesk” where
such problems should be reported (an “intake”). They avoid using it, out of general fatigue,
believing it will take up a lot of their time and not solve their issue. We confirmed that there are
5
“storage area network,” which means “expensive disk drives.”
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p29
several escalation layers that have to review and pass up something like a mainframe
performance issue. It was unclear whether anyone had reported the performance issue that we
saw firsthand.
Some work queues contain hundreds of thousands of small items that are expected to be quick
and easy to resolve. If each flag flip takes several minutes to “sync,” the productivity loss can
be severe.
Recommendations:
CUBS-SCDB is a textbook example of a system “seam” issue that nobody owns. The many staff
dedicated to CUBS and its infrastructure can’t see if there is a performance issue on SCDB, and if
there is, they can’t do anything to fix it. This leads to a situation where even if there is a CUBS
issue, it’s not likely to be detected or fixed, because it is easier to say it’s SCDB’s fault. (We
didn’t see any reason to believe there are such issues in CUBS, to be clear. It is just a standing
risk.)
On the other side of the mirror are the SCDB and mainframe staff, who don’t have any metrics
to tell them that CUBS performance is slow, and have little incentive to create any.
A higher tier of the organization, or a common stakeholder, needs to lead a process to define the
SLA
6
for CUBS sync performance. Then the SLA needs to be instrumented with a monitoring
tool that is clearly and easily accessible to everyone from the line worker to the mainframe
operators in ASD. Then, even if the multiple escalations through IT Helpdesk are left as-is, it
will be possible for real problem reports to move expeditiously up the stack.
System Monitoring
It is critical that system monitoring correctly reflects the experience of the end user. As
described above, we observed field claims workers frustrated by the performance of SCDB. In
their words, “the mainframe was down” and “syncing took 20 or more minutes.” From the
view of the field office, work had stopped, and they filed multiple tickets with the ITB helpdesk.
6
“service level agreement”
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p30
When we independently escalated to IT, we were told that there had been some small
performance issues that day. No one in IT realized the impact on the field.
Claimant experience should be monitored in the same way. Facebook groups and legislative
emails are full of reports of strange UIOnline errors and potential bugs. The “Voice of the
Customer” report offers data on how people are interacting with EDD. These sources should be
used to uncover system problems which can then be monitored systemically.
Recommendations:
● We recommend that IT tighten automated alerting conditions such that they learn about
end-user outages before tickets are filed. Looking back at data from prior outages can
help define these thresholds.
● UI Leadership, including staff from ITB’s Product Development Division, should
regularly review the “Voice of the Customer” reports to locate problems with the
claimant experience which aren’t caught by other statistics.
● In addition, the claims offices need a direct escalation path to ITB. Their issues can still
go through the helpdesks, but they need to carry information describing the impact of
the issue in the field office. There will always be disagreements about the severity of an
issue, but should not be possible for one part of EDD to believe it has a “minor
performance issue” while another part of EDD believes “the mainframe and the call
center are down.”
Employees are managing passwords insecurely.
EDD employees must juggle many different systems throughout the day and are struggling to
keep track of passwords.
Recommendations:
● Procure an enterprise-wide password management tool for employees.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p31
Some mailing addresses are too long to fit in the database.
When a claimant or employer has an address that is too long to fit in the database, it must
trigger a manual claim. This also means that all international mailing addresses are routed for
manual review simply because the address needs to be modified to fit in the field.
As of August 29, 2020, the following work items were pending manual completion:
● ‘Truncate Address’ (6,142)
● ‘Truncate Employer Address’ (1,552)
● ‘Standardize Address’ (1)
From May 17, 2020 to August 29, 2020, there were 24,925 UIOnline visitors who reached the
new claim confirmation page from Mexico. Historically, 1,232 claims have been diverted to
manual processing solely for having an international mailing address that needs to be
shortened by a human.
Recommendations:
● EDD IT should develop a plan to accommodate addresses of any length and cease manual
review of claims solely because they have long or international addresses.
● This solution should result in turning off CPT Code 023, “Claimant entered an
international mailing address,” and removal of CUBS work item queues “Truncate
Employer Address” and “Truncate Address.”
Scans of submitted identity verification documents do not include the color red.
When the identity verification office receives documents, it sends them to a scanner which
cannot scan the color red. Color copies of California driver’s licenses, which have date of birth
and ID numbers printed in red, lose this information once scanned. As a workaround, the
clerical staff in the identity verification office must hand write any red elements on the
document before sending it to the scanner. This slows down processing and introduces an
opportunity for error; we witnessed at least one handwritten number that transposed “1” and
“7”.
Recommendations:
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p32
● RESOLVED: The scanner was able to adjust and scan the color red, and the identity
verification clerical staff no longer hand-write on documents.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p33
Business Process Obstacles
No systematic, data-driven way to define and work down the backlog.
As stated in the Executive Summary, the only way out of this unbounded growth is ongoing
measurement and action. A dashboard of pending unemployment claims with clearly explained
definitions of each number and category, commonly understood by key stakeholders, is
foundational. This has also been requested by members of the legislature in a letter dated
August 5, 2020.
Through our work with EDD, we came to understand their existing top measurement indicators,
which are:
● Making the first payment to a claimant who is entitled to a payment within 21 days (10
days for Work Share claims). This is known as Regular Unemployment Insurance First
Payment Promptness, and the goal, set by the U.S. Department of Labor, is 87%.
● Making 80% of non-monetary determinations within 21 days.
○ In the quarter ending June 2019, CA EDD was at 64.7%
○ In the quarter ending June 2020, CA EDD was at 17.1%
● Staying under a 10% improper payment rate, which is required by U.S. Department of
Labor. UI Improper Payment Rate data is derived from the Benefit Accuracy
Measurement, a sampling of up to 480 cases from California.
○ California has met this measurement year after year; from July 1, 2016 through
June 30, 2019, California’s improper payment rate was 8.019%
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p34
However, none of these indicators resonated for capturing the current pandemic workload. In
particular, filtering out those who received first payments would miss swaths of constituents
who received an initial payment but have unresolved issues with their claim.
This view also does not explain the drivers behind the timeline (or missed timelines), in order
for EDD to take a data-driven approach to resolving those issues, starting with the largest ones.
Overall, we have seen little evidence of EDD using data to make decisions.
Recommendations:
● EDD and the Labor Agency should adopt the new definition of the backlog, developed by
the Strike Team with EDD’s input, described in Appendix A: Data-Driven Definition &
Resolution of the Backlog.
Update: EDD has adopted this definition.
● EDD IT staff should run the reports needed to update the backlog dashboard daily. These
reports have been developed by the Strike Team and handed off to EDD staff. CDT has
developed the dashboard and is in the process of handing it off to EDD staff.
● EDD should establish a semi-weekly “incident command” update.
○ This should focus on a recent (less than 24h old) refresh of the “backlog”
number and its contributing metrics.
○ EDD staff should provide brief (less than 1min each) updates on in-flight
short-term suggestions, to include any blockers for senior leadership to
immediately clear.
○ Progress: This incident response team was established on August 28, 2020,
chaired by EDD’s Carol Williams.
UPDATE: The above are all in progress as of the date of this report.
EDD is not consistently using its own centralized Document Management (Mail,
Scanning) Services
Across field offices, we witnessed inconsistent mail and scanning practices. Every office we
visited received mail directly, including multiple offices within the same building. The identity
verification office received and sorted its own mail, but drove it to the document management
center for scanning. Another office scanned its own mail, sometimes, on a document scanner in
the corner. Another office had an employee pick up a box of appeals mail to process on
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p35
Wednesdays. Yet another office had no scanning at all. In every office, we saw significant
unopened mail, and significant mail that needed to be re-routed (e.g. identity verification
documents sent to an office that was not the identity verification office).
EDD has a central document processing center. It has the capability to receive mail in one
location using different PO Box numbers to distinguish between offices. And it has the capability
to open, sort, and scan that mail in one place. But it is not using it.
Recommendations:
● In the short term, redirect new employee hires to fix returned mail. (Attempt to resolve
the address with USPS, resend it, log the correction in the system.) In the long term,
procure a solution to automatically resolve returned mail for the central document
management facility.
● In the short term, drive or bulk ship all mail, unopened and unprocessed, to the
document management center on a daily basis.
● Move all mailing addresses to PO boxes under the single central document management
center
● Change all return addresses on all envelopes to go to the central document processing
center
● Scan all mail. Offices that do not currently use scans, like Appeals and Work Share, can
follow the same initial plans as Identity Verification, which switched to working off
scans in under two weeks.
● Count all mail, by type (which can be determined by PO Box number), and include this in
backlog reporting as a leading indicator.
● Track days to open/process the mail, with a goal of opening all mail within one day.
● Until moving to the central document processing center, weigh unopened mail and
report the weight from each office each day. The goal is not to have zero pounds of mail,
but rather to ensure volume of unopened mail does not grow. (Based on weighing
samples in the field, our baseline numbers are 450lbs of unopened mail per field office,
with approximately 15lbs of appeals per day. The strike team has already provided some
offices with scales and training on weighing the mail.)
● This recommendation does not apply to internal, interoffice mail.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p36
The backlog of recomputation tasks is growing.
We heard significant interest from EDD in helping reduce/automate the backlog of
recomputation tasks, specifically. Recomputation bears the brunt of all manual processes,
including manual identity verification and manual wage verification. For example, one brief
manual identity verification work item triggers at least three somewhat lengthy recomputation
work items.
The growth of this workload is staggering. In the year from July 2018-July 2019 recomputations
completed 140,520 work items related to unemployment. This year, in July alone, more than 6
months of work (82,067 items) were generated for this team. And this tracked recomputation
inventory does not account for the many tasks that the recomputation team must do as a result
of legislative escalations and call center review assignment sheets. We estimate this additional
set of work items exceeds 60,000 work items so far in 2020.
Recommendations:
● (As above) Our analysis shows that most recomputation tasks are a result of manual
identity verification, and that moving to an automated identity verification service will
also stem the tide of recomputation tasks.
● (As above) It is particularly critical in recomp to reassign new hires to tasks such as mail
processing, as new hires in recomputations have no skills-appropriate work, and
experienced staff are unable to complete recomputation tasks while training and
managing new employees.
Claims processors are unable to return work items that they cannot resolve back
into the pool.
We observed multiple claims representatives who cannot return a work item to the “pool” for
someone else to work. For new recomputations employees specifically, a supervisor must
reassign individual work items in BCIS to other employees, and they are receiving thousands of
such reassignment requests each day. This is a large and unnecessary strain on management,
and also causes delays in claim processing times when work items are “hidden” in individual
workers’ queues.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p37
Recommendations:
● Automatically move BCIS/CUBS work items back into the pool after 24 hours of
inactivity.
● Enable all BCIS and CUBS users to return work items back into the pool.
Claims processors cannot always access the necessary skills/queues to resolve a
claimant’s issue in one session.
A central Support Division team assigns specific “skills”/queues to individual employees.
Experienced employees report that they can often see an open work item on a claim they are
working, which they know how to resolve, but they do not have permission to do so.
This restriction is in part due to the required “separation of duties” between claims filers and
claims payers, to protect against internal fraud. However, given the circumstances, even those
restrictions may warrant temporary elimination, with future auditing and consequences should
an employee be found to be generating and paying on false claims. A top request we heard was
for all EPRs to be able to change the effective date on a claim (backdate it).
However, not all restrictions of queues are due to this separation of duties. In some cases, tasks
are simply overly-segmented across the workforce, and the restriction does not take into
consideration whether an employee was previously trained in another area (e.g. Work Share).
A related observation is that even long-tenured EPRs sometimes don’t know how to process an
item that is within their assigned “skills.” They routinely re-route such items to someone else
that they know, or simply stand up and go to another cube to ask what to do. In some cases, the
process has become institutionalized in the form of large Excel spreadsheets that are circulated
among the field office managers.
These “off the books” workarounds are commonplace, and they enable progress, but they are
invisible to the Support Division, who are left believing the “skills” and “separation of duties”
segmentations are working as intended.
The general consensus in the field is that EPRs should be given the ability to access all pending
work items, and if they actually don’t know how to complete one, they would not. The Strike
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p38
Team supports this position based on our observations. In the near term, it should be sufficient
to trust the senior EPRs to know what they can and can’t handle.
Recommendations:
● EDD should vastly simplify the “skills” segmentation of work queues, enabling
experienced EPRs to act on all the items that they know how to handle (and only those
items).
● The Support Division should assign all skills and all queues to every EPR (not EPR-LTs).
● EDD should reconsider the segmentation that has been created for “separation of
duties,” and retain only the ones that are truly necessary for a statutory reason.
● In the long term, institute a quality control program (such as having a small sample of
each EPR’s work product reviewed by another senior EPR); this would be more effective
than the current system.
People have multiple open claims at once, and they should not.
272,103 people filed more than one claim between 04/28/20 (the day PUA was implemented)
and 08/01/20, even though you should only have one open claim every 365 days. (52,636 people
filed more than one claim in July alone.) This causes significant rework and further delay, as
every multiple claim has manual work items attached to it, and the claims must be manually
merged in order to decide the claim. This manual work is handled by the recomputations team.
We have confirmed that you can only have one claim open every 365 days, even across UI, PUA,
extensions, WS, and multiple jobs/periods of employment. Even if you currently have a closed
claim from within the last 365 days, regardless of whether you have a new period of
unemployment or you are switching between UI and PUA, you would “re-open” it, not file a
new one. (Technically this could be an “additional claim” or a “reopened” claim depending on
whether the claimant worked in-between certifications - but it would not be a “new” claim.)
People are likely filing multiple claims due to confusion; they may not have heard back on their
original claim, and so filed another one. According to the Voice of the Customer reports,
upwards of 100 people a day have said “reapply” or “apply again” to the chatbot since July 20.
Allowing people to file multiple claims is adding significant burden to the workforce and further
delaying claim processing times.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p39
July counts Breakdown:
Number of Claims filed
by one claimant
# of Claimants
2
45615
3
5011
4
1142
5
390
6
181
7
102
8
46
9
39
10 or more
110
Total count of claims filed on/after 4/28/2020 (PUA implementation)
Number of Claims file by
one claimant
# of Claimants
2
220705
3
35767
4
9357
5
3220
6
1267
7
651
8
356
9
207
10 or more
573
Recommendations:
● Hide the new claim button in UIOnline if the user already has an existing claim from the
last 365 days.
Out of state wage verification is manual.
While some wage verification can be resolved through inter-state data systems, proof of wages
from out of state from the last two years often must be provided by claimants. Since the new
document upload feature is not yet available for wage documents, claimants must mail in proof.
Approximately 4% of claims involve out-of-state wages.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p40
Recommendations:
● Expanding document upload to include non-identity documents will at least speed this
process up.
Claims from newly-separated Servicemembers (Veterans) cannot be filed
online, only on paper.
Newly separated military (people who are now veterans) are eligible for unemployment for
more reasons than other citizens, due to federal law. These veterans cannot file for
unemployment online in California. There are currently 11,604 pending Veteran claims in CUBS.
They must provide EDD with the specifics of their federal employment on paper via Form DE
1101IAD (including SUPPLEMENTAL FORM FOR EX-SERVICEMEMBERS – ATTACHMENT A).
The only difference in the process is this ATTACHMENT A.
EDD uses the veteran’s Form DD214 as the final and conclusive proof in terms of eligibility
(based on character of discharge - you must have an honorable character of discharge) and
monetary award (based on last pay grade). This information is found on Form DD214. Form DE
1101IAD SUPPLEMENTAL FORM FOR EX-SERVICEMEMBERS – ATTACHMENT A allows a
veteran to submit their DD-214 in lieu of ATTACHMENT A, but if they do not submit it, EDD will
call them and ask them to mail it in. The form does not make this clear.
Recommendations:
● Replace SUPPLEMENTAL FORM FOR EX-SERVICEMEMBERS – ATTACHMENT A paper
form with a requirement to attach a copy of their DD-214. EDD will request the DD-214
anyway; this eliminates that extra step.
● In the short-term, as soon as Document Upload for all documents is available, allow
Veterans to file electronically. This would be the normal UIOnline form, plus a
requirement to upload their DD-214.
● In the near-term, EDD should fully automate military claims using the Veteran
Verification API from VA.gov to automatically confirm DD-214 information directly from
the federal government. The data elements needed from the VA are available in this API
(under service_history), and it is free to use. Since EDD and VA.gov both use Id.me, there
may be additional efficiencies or reusable code already available for this.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p41
Federal civilian employee claims cannot be filed online, only on paper.
Former federal government employees cannot file for unemployment online in California. They
must provide EDD with the specifics of their federal employment on paper via Form DE 1101IBD
(including SUPPLEMENTAL FORM FOR FEDERAL EMPLOYEES – ATTACHMENT B). The only
difference is this ATTACHMENT B.
ATTACHMENT B asks for the exact three-digit Federal Identification Code (FIC) of their former
federal duty station, which is the key information needed to process federal civilian employee
claims. Field representatives reported was very difficult for people to do accurately, and usually
involves a back and forth of phone calls, despite this information being clearly printed on the
SF-8 form every federal employee is issued at separation. Since all that is needed is the FIC, it
would be easier and more efficient to simply have claimants send their SF-8 forms instead, and
let the claims processors enter the number from that form into the system.
Recommendations:
● Replace SUPPLEMENTAL FORM FOR FEDERAL EMPLOYEES – ATTACHMENT B with a
request for an SF-8 or SF-50. Federal employees can use the normal electronic UIO form
with the addition of instructions to upload their SF-8 or most recent SF-50.
Stale work items, some as old as 2013, obscure current claims that require work.
Many open work items, particularly in CUBS, are no longer applicable. Leaving them open in the
system makes it harder to identify which items do require work, while also making the total
number of pending items inaccurately and alarmingly large.
For example, due to the aforementioned challenge that claims representatives cannot return
work items to the “pool,” some work items from 2013 or earlier remain open in CUBS, in
individual representatives’ work queues. Work items this old cannot be relevant any longer; an
individual claim can only be open for a 365 day period. This clutters individuals’ queues, and
these obsolete work items would count towards the total number of work items in the system.
This is an inaccurate representation of the actual work item inventory.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p42
Additionally, our investigation found that some work items are merely automated system
notifications that can safely be ignored. Some of these work items go by names that sound
pressing, which is confusing. For example, a work item called “Stop Payment Alert - Claim
Review” has 730,023 open work items as of 08/29/20. This seems alarming at first glance.
However, field representatives informed us that this work item is merely an automatic,
system-generated notice when an actual Stop Payment Alert is placed on a claim. The work item
itself is not a Stop Payment Alert, and it is a leftover system notice on an estimated hundreds of
thousands of claims. A query for work items beginning with “Stop Payment Alert” on claims
that do not have a Stop Payment Alert would clean up these categories and accurately reflect
pending work load.
Recommendations:
● Delete work items that are over 365 days old. (Going forward, the solution to this
problem is referenced earlier: automatically returning stale items to the pool.)
○ Status: In Progress by EDD as of 08/28/20
● Delete any work items beginning with “Stop Payment Alert” if the associated claim does
not have a Stop Payment Alert on it. Run this on a weekly basis.
○ Status: In Progress by EDD as of 08/28/20
New employees should be reassigned to skills-appropriate work until the
backlog is vastly reduced.
As explained earlier, it normally takes at least six months for a new EDD claims representative
to be able to do useful work, and even then their allowed task set is limited. With the shortened
training time, as explained earlier, it appears they can do even less. New employees are lowering
the overall capacity of EDD to less than it had before the pandemic.
For our recommendations, see the appropriate section under Existing Issues/Challenges.
The EDD Legislative Office does not have a tool to track legislative requests.
Before the pandemic, EDD’s Legislative Office received approximately 1,000 unemployment
insurance related inquiries each year. Since the beginning of the pandemic, they have in some
cases received 1,000 cases per day, with a current backlog estimated at 16,100 unprocessed
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p43
requests, presumably because claimants call their legislators when they can’t get help from the
EDD call centers (see section on call centers above.) Dissatisfied legislators have requested that
experienced claims processors be assigned to work with their offices directly.
In the creation of the backlog definition, all stakeholders have agreed that resolving constituent
complaints raised to their legislators is a high priority for the Department. When we met with
the EDD Legislative Office to incorporate their data into our dashboard instrumentation, we
learned they have no tracking tools at all -- individual constituent concerns are handled
individually by email.
This means there is no data to incorporate into the dashboard, but also that some constituent
concerns are likely going unresolved -- and also that a lot of constituent personal information
is being transmitted by email, which contributes to information insecurity.
Recommendation:
● Get a case management / CRM system for the EDD Legislative Office in the next two
weeks. See Appendix C: Acceleration Plan for Legislative Case Management for our
acceleration plan.
○ UPDATE: This is now available and will be launched soon, pending coordination with
the Legislature.
Order of payments on CUBS screen is the opposite of Work Share Form DE
4581WS
Work Share Form DE 4581WS lists two weeks of payments. On the CUBS payment screen, the
order of payment weeks displayed is the opposite order of the form. We had multiple employees
who process payments tell us this is a common cause of payment errors. While we do not have
data to substantiate how often this error occurs, it makes sense to us that the flipped order
could cause issues.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p44
Recommendations:
● Flip the order that payment weeks display in CUBS.
Claims Representatives Are Not Allowed to Email Claimants
We heard from management and field representatives that employees cannot respond to
claimants by email (or initiate new emails) because it would expose their individual EDD email
addresses. We appreciate that exposing any individual or group email address right now does
invite a high number of incoming messages that cannot be processed, and that staff time is
better spent working open claims.
Recommendations:
● Encourage use of Web Correspondence, which enables two-way communication through
the UIOnline interface. This is not perfect due to the high volume of messages that may
make it hard to see a reply right away, but is an acceptable interim step. EDD IT should
consider this use case in development of its future-state customer service platform.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p45
Gaps in End User Experience
Claimants do not understand their status
“Claim Status” is a top reason for contacting EDD. Neither the claimants nor the call center
operators understand the status of claims, which leads to a growing number of confused and
dissatisfied customers further straining the system.
As of August 29, 2020, there are 399,722 pending electronic messages from claimants
categorized as “Filed a Claim - No response” and 211,876 categorized as “Where’s my
payment?” As of July 2020, at least 20% of chatbot requests were related to claim status. And
claim status is the top reason for calling the call center.
In a three-day pilot from 7/27/20 - 7/29/20, the Office of Digital Innovation added Google
Analytics to track how often people are clicking the help icon when using UIOnline. There were
338,753 help button clicks during this three-day period. Of these, 77,750 were users trying to
understand their claim status.
EDD attempted to address this confusion in a July 30, 2020 press release, which outlines the
possible claim statuses available today:
Helping claimants understand their claim status
The EDD continues to enhance communication channels with claimants to help them better understand the
status of their claims. Help text is now provided at the top of the “Status” column on the home page of
their UI Online accounts as work continues to further develop help text as part of ongoing efforts to
enhance the claimants’ experience with the UI program. Some of the most common status updates on a claim
include:
● Paid – claimant met all eligibility requirements for receiving a bi-weekly payment.
● Pending – claimant responses to certification questions indicate a potential issue that had to be
directed to EDD staff for review.
● Disqualified – claimant is not eligible for payment for this week(s). The claimant will receive a
Notice of Determination in the mail that will identify the specific reason for the
disqualification and explains the right to file an appeal if the claimant disagrees with the
disqualification. Claimants should follow the instructions on the notice about how to appeal. .
● Excessive Earnings - the earnings reported, when reduced by $25 or 25%, are greater than the
weekly benefit amount; therefore, no benefits can be paid for this week.
● False Statement Penalty Week - The week being certified cannot be paid and will be applied to a
false statement penalty. The penalty was most likely assessed on a prior UI claim when the
individual may have returned to work but failed to notify the EDD and continued to collect
benefits at the same time. This is the most common type of fraud that results in a
disqualification for the benefits originally paid and a penalty. For more information see
Overpayments and Penalties on the EDD website.
A claim status of “pending” is too broad. Nearly everyone that is seeking to find out their claim
status will get the answer of “pending.” Claimants need specific status updates that tell them
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p46
where their claim is in the process, what the next step is, who has the next step (them or EDD),
and how long to expect to wait.
Recommendations:
● Scope and implement a solution to meet the needs of claimants in knowing their status
○ In progress: EDD is working with an external vendor, Nava PBC, on a claim status
feature.
Claimants do not understand the form questions and make mistakes
We consistently heard from field employees and from claimants themselves that some
questions on the initial application and on the recertification form are confusing, and even if
answered “correctly,” they are driving a great deal of concern, stress, and contacts with EDD.
EDD addressed this issue in its July 30, 2020 press release:
How to avoid common mistakes
Unemployment claimants are required to remain able and available to accept suitable work if it were
offered in order to be determined eligible for ongoing benefit payments. The usual requirement to look for
work each week is temporarily suspended during the conditions associated with this pandemic. The EDD has
found many claimants are having issues answering the first two questions on the required bi-weekly
certification and has added help text in the UI Online application as well as in the step-by-step UI
Claims guide on the EDD website to better assist claimants.
● The first question asks “Were you too sick or injured to work.” While there may be COVID cases
complicating the workplace, if you were not sick and were able to work if offered, you should
answer NO on this question to ensure you have remained able and available.
● The second question asks “Was there any reason (other than sickness or injury) that you could not
have accepted full-time work each workday?” Again, if you had remained able and available to
accept work if offered, you should answer NO to this question.
● Answering YES to these questions could disqualify you for benefits for that week, requiring the
EDD to review your claim for eligibility and schedule a phone interview with you to collect more
information if needed.
In addition to the regular state Unemployment Insurance program, the EDD administers the separate Pandemic
Unemployment Assistance (PUA) program for those who don’t qualify for regular UI including business
owners, the self-employed, independent contractors and others. These benefits are paid for by the federal
government and the EDD is required by law to ensure claimants meet specific COVID-related reasons for
their unemployment or reduced hours before issuing payments.
● The EDD has discovered many PUA claimants are running into a disqualification for benefits based
on the way they are completing their application for benefits through UI Online, or completing a
notice sent through the mail asking them to verify why they are unemployed. There are several
reasons provided, including your place of business has closed due to COVID or the pandemic has
severely limited your ability to perform your customary work. If claimants select the last option
of “none of these apply”, then the EDD is required to deny benefits. See PUA FAQs for more
information on eligibility requirements.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p47
Even assuming perfect comprehension of this guidance, this leave claimants who have been
offered work and are not sick but fear contracting COVID-19 unclear about what how to answer
without disqualifying themselves. Because the backlog in claims is widely known, many
claimants are aware that a required phone interview likely means they will not see their claim
processed in a reasonable time frame, leaving them distressed about to answer the questions
accurately while retaining hope of receiving the benefit.
We heard a few objections to updating the existing forms. One was equity; EDD did not want an
improved digital form available without also extending the improvements to those who file on
paper. A second was the difficulty of updating either the website or the scannable paper forms,
and certainly the time to update both. The third was the underlying database changes that may
be required, both in order to differentiate who answered the “new” version of a question vs the
previous kind, and in order to accommodate new/modified data elements.
We appreciate these challenges and agree that an overhaul of all the forms is a significant
undertaking. But as a matter of practical reality, 95% of all initial claims are filed electronically;
it is acceptable to improve the experience for 95% of users first by tweaking language and help
text in the electronic version and, once data proves those changes were positive, to port them
over to the paper form soon after.
In particular, the two questions referenced above in EDD’s press release (“Were you too sick or
injured to work” and “Was there any reason (other than sickness or injury) that you could not
have accepted full-time work each workday?”) are causing extreme confusion and stress
amongst claimants because it is unclear how to answer them in relation to the COVID-19
pandemic. We also understand that while EDD is automatically processing recertifications
regardless of the response, they will eventually have to manually re-evaluate millions of these
recertifications in the future. Managers informed us that the question of whether you are
physically able to work continues to be adjudicated manually, which is likely still causing
confusion since COVID-19 is a health-related issue.
Another common case of confusion was reporting net vs gross wages, including some situations
where this discrepancy could have resulted in a claimant being charged with Fraud Statement
Weeks. Reporting net wages instead of gross wages would result in a lower payment, so it is hard
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p48
to see why this would be considered fraudulent activity, but regardless, EDD does not appear to
have a clear process.
In a three-day pilot from 7/27/20 - 7/29/20, the Office of Digital Innovation added Google
Analytics to track how often people are clicking the help icon when using UIOnline. There were
338,753 help button clicks during this three-day period. Of these:
● 101,112 (one third) were on the certification form
○ 13% did not understand what they were “accepting” on the form.
● 100,110 (another third) were on the new claim application form
○ 26% of the users who were confused by the questions “What type of work do you
normally perform?” and “*What other type of work can you perform?”
We heard repeatedly that form improvements were on hold because they would be addressed by
the BSM project.
EDD’s continues to add more instructional language/pages/guides for claimants in the hopes
this confusion will abate (referred to as “doing a better job of education”) and certain recent
efforts have helped. Google Analytics data shows that users who read the new plain language
claim guide had 2-4x the rate of success filing a claim than others. But overall, this has had
limited impact. It is not possible to force claimants to read a manual before they start, and it is
not reasonable to assume that they will then somehow keep it open in another window and
bounce back and forth between the manual and the application all the way through the process.
Requiring they do so to be successful disadvantages
They need intuitive, guided interactions with contextual help, with a single and brief plain
language user guide.
Recommendations:
● Adopt a data-driven approach to continually improving the claimant experience,
described in greater detail in Appendix D: Plan for evaluating and further iteratively
improving the claimant experience.
● Assign WCUG to work with real users to develop new language for the first two
recertification questions above with real users, and implement the new language to the
online form within two weeks. As part of the implementation, measure the use of the
help icon next to each question via Google Analytics, relative to the time period prior to
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p49
the new language, with a goal that a smaller percentage of people require help
answering the questions.
● In parallel to changing the online language, make a plan to release a new version of the
paper form two weeks after the digital version (to allow time to validate the new
language makes more sense to users).
● Work with WCUG to develop a new, “ideal” unemployment claim initial application form
and recertification form. This form should be developed primarily with real end users,
with subsequent input from legal and technical teams. They should work closely with
advocates and researchers to understand user needs. Working backwards from this ideal
claim, EDD should develop an implementation plan of what changes can be made
immediately or in the short-term, and which would require significant restructuring.
This should be a collaborative effort, with each resulting revision tested with real users.
● Explore whether claims with CHO == 850 (Work Share) could be automatically prevented
from recertifying online, with an instructive error message (that they cannot recertify
for UI because they are on a Work Share plan).
Claimants receive “zero dollar award” letters they do not understand.
Today, if a claimant is sent to identity verification, is a military claim, is a federal employee
claim, or requires out of state wage verification, their wages are “suppressed” (hidden to avoid
disclosure to someone who may not be the claimant). This means when they receive the
automated “Notice of Uninsurance Award” notice, it shows $0 in benefits. This letter has
caused many frantic, repeated calls to EDD and legislators that could be avoided with clearer
messaging.
Since CUBS cannot track the suspense period from the date a notification is mailed, there is also
no way for claimants to track via UIOnline (or in CUBS) when responses are due.
Recommendations:
● Do not send the “zero dollar award” notice when wage information will be suppressed
● Clear all public notifications and individual notification letters (mailed, SMS, and
electronic) through the Strategic Communications office to ensure consistent language
and to catch opportunities for confusion like this.
● Obtain a level of effort to include notifications in a claimant’s UIOnline account.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p50
● Obtain a level of effort to include due dates with notifications in a claimant’s UIOnline
account (and make them visible in CUBS).
Claimants must mail in their documentation.
Claimants have no way to upload documentation - wage verification, DD-214s, SF-8s, etc. The
only way to submit documentation is to physically mail it.
This is also the reason why military and federal employee claims must be filed on paper.
Recommendations:
● Implement the plan outlined in Appendix E: Acceleration Plan for Wage Verification.
● Expand use of the new document upload tool for all documents; make it work on mobile
devices; and enhance its usability with the WCUG team.
The EDD UIO website does not work on mobile phones.
The Office of Digital Innovation’s research showed 39% of users in June were using mobile
devices to file a new claim in UIOnline. The regular EDD website has closer to 70% of its users
coming from mobile. With the introduction of document upload, most users will presumably
photograph their documents with their phone, leading to increased mobile users.
The Office of Digital Innovation has already written code to make the UIOnline form work on a
mobile phone, but was told the level of effort from Deloitte to implement it would exceed eight
months. While we appreciate the fragility of the UI Online website code, we find this estimate to
be unacceptable and inaccurate.
There is a separate “UIO Mobile” application, with extremely limited functionality. For most
tasks, the link takes you back to the desktop site, which does not work on mobile.
Recommendations:
● Work with ODI to replace the current UIO application template with a mobile responsive
one within the next 30 days.
● Turn off the “UIO Mobile” application at the launch of the mobile responsive template.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p51
● Do this in concert with the below recommendation to make the UI Online application
WCAG 2.0 compliant.
A striking number of people are recertifying on paper and by phone.
[Update: this issue was resolved since starting this report. In August, 92% of certifications were
received electronically.]
While more than 90% of claimants file their initial claim online, 40% recertify on paper, and
8% via telephone (via Tele-Cert). Increasing the number of electronic recertifications lowers
the workload for EDD, at the very least in mail processing.
It is hard to get specific data on why so many claimants revert to paper, but our research
revealed some themes.
● Since the website is so difficult to use via mobile phone, claimants are willing to navigate
it once to apply, but find the paper easier going forward.
● Claimants get a paper certification form in the mail along with their award notice, and
users continue receiving paper forms unless they opt-out. It is less friction to fill out a
paper form right in front of you than to go online and do it. (96.7% of users opt out of
paper notifications.)
● Claimants stuck between BPO and UIO who are unable to complete UIO registration
cannot recertify online.
● Many users report through social media getting 404 errors or having difficulty logging
in to recertify.
● People with partials, Work Share, and certain other types of claims can only recertify on
paper.
Recommendations:
● Change the “mail by” date on paper recertification forms to be two weeks farther into
the future. Claimants often receive this form after the “mail by” date and worry it’s
already too late.
● Work with WCUG on user research to increase the online recertification percentage to be
90%.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p52
The usability team is last instead of first in the development process.
The usability team is not included in the development process until after the requirements have
all been finalized and the new changes are ready for user acceptance testing. This is too late.
EDD is missing out on the chance to make user-friendly, usable services for claimants. The
current UIOnline offering is not easy to use.
Recommendations:
● Include the WCUG usability team in the very beginning of the development process,
prior to development of requirements.
● Provide a testing environment separate from UAT that is available to the public (outside
the EDD network). This is the only way EDD can test new features with actual end users.
The EDD UIO website is not accessible to those with disabilities.
California Assembly Bill 434 requires all state websites to be “in compliance with Sections 7405
and 11135, and the Web Content Accessibility Guidelines 2.0, or a subsequent version, published
by the Web Accessibility Initiative of the World Wide Web Consortium at a minimum Level AA
success criteria.”
EDD is also held to federal 508 compliance standards requirements.
The EDD website is out of compliance with the above and entirely unusable with modern screen
readers and other accessibility tools. This means Californians with disabilities cannot file or
certify claims online (including disability claims).
Recommendations:
● Move the involvement of the Web Content and Usability Group (WCUG) and the Digital
Accessibility Team to the beginning of all development planning.
● Conduct a WCAG 2.0 audit of the UIO website and address all findings. This should begin
once other key recommendations regarding the backlog are well underway.
● Update the UIO website to be WCAG 2.0 compliant (addressing issues identified in the
audit above). This should ideally be done at the same time as making the UIO website
mobile responsive, since most of the changes will fall under both categories.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p53
● Require that all members of the development team use the UIO website via an
accessibility tool (e.g. screen reader) during User Acceptance Testing. It is possible to be
technically compliant but still be so difficult to use that a user would need three or four
hours to navigate a single form using a screen reader; such a result would not be
acceptable.
● EDD must include representative end users in its user acceptance testing going forward,
including users with a wide range of abilities and accessibility tools.
Individuals who are not fluent in English face insurmountable barriers to
receiving assistance.
Claimants who speak languages other than English have a difficult, if not impossible, time
navigating the unemployment application process.
An August 21 2020 letter to Governor Newsom from Senator Holly Mitchell's office highlights
some of these concerns:
“EDD is potentially in violation of the Dymally-Alatorre Bilingual Services Act, which requires every
state agency “directly involved in the furnishing of information or the rendering of services to the
public whereby contact is made with a substantial number of non-English-speaking people, shall employ a
sufficient number of qualified bilingual persons in public contact positions to ensure provision of
information and services to the public, in the language of the non-English-speaking person.” Californians
for who English is not their first language face tremendous barriers in the EDD process, and call center
staff and applications for programs offered by EDD are not available in enough languages. EDD should work
with other agencies to leverage existing language resources and prioritize language accessibility.”
We surveyed field staff to understand what happens when someone needs language assistance.
We learned that EDD does have access to Language Line, an on-demand translator service that
lets you dial a live interpreter into a call, but it requires managerial approval to use each time.
This is hard to get in real-time with a newly-distributed remote workforce.
Before the pandemic, we were told that the manager of each field office was responsible for
identifying a staff member who could speak an individual constituent’s language, but that this
could take hours.
While “preferred spoken language” is captured on the unemployment application, it did not
seem clear to representatives whether a claimant required a translator, or what to do next if
they did.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p54
Recommendations:
● Allow EDD representatives to use Language Line to call out to claimants without
managerial approval.
Users are regularly providing the wrong identity documents.
If a claimant is required to verify their identity, they must send back (in the mail) two forms of
identification from an approved list. Enough users find this list confusing that 56,256 people so
far have sent in identity documents that are not on this list. Of those, 11,404 were rejected and
disqualified because of it.
Early usability testing of the new document upload feature also shows that test users -- who, it
should be noted, are EDD employees -- were also uploading “incorrect” documents.
This confusion is exemplified by Identity Verification help page on the popular website FAQ CA
UI, which shares the following story of a user sending in incorrect documents as an example of
what to do:
When I was finally able to get through on the phones a few days ago to resolve the incorrect
SS# (system glitch?) on my original claim filed in early April, the representative was able to
put that claim back in motion (again, 10 days for the letter, etc.) but because I still have an out
of state drivers license (I've lived here three years), she directed me not to wait and to go
ahead and make a photocopy of my drivers license, and proof of residence/address (most
usually a copy of a utility bill, in my case I sent in a copy of a bank statement, AND my
stimulus check—not on their list, but it seemed like a good idea), to write my name and
social on all the pieces and include a letter explaining and asking them to "please process the
documents."
Recommendations:
● This issue will be resolved at the launch of Id.me.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p55
Claimants are waiting on EDD for information that only Bank of America can
give them.
The seventh-largest work item queue is electronic messages related to the EDD Debit Card (the
CUBS queue called “WC-EDD Debit Card”). As of 08/29/20, this queue has 352,236 pending
work items.
EDD cannot help with debit card questions. Only Bank of America can.
Additionally, Bank of America is telling claimants who have not received their debit card yet
that they can pay a $10 fee to expedite delivery of their card. This fee is exploiting Californians.
Recommendations:
● Do not accept “WC-EDD Debit Card” messages in UIO. EDD cannot resolve these.
Instead, when a claimant tries to select this option, redirect them to BOA’s website and
phone number.
● Ask Bank of America for a daily dashboard of their call volume, call resolution rate, hold
times, and pending issues by category.
● Disallow Bank of America from charging the $10 expedited fee.
Errors in SSN data entry may be common.
In the online application, users must enter their social security number, then confirm the last
four digits. By default, the social security number is obscured by asterisks (*** ** ****), with
an option to check “Unhide” to view the actual number.
It is difficult to tell from Social Security Administration error codes how often people are simply
mis-typing their SSN. However, this problem may be more pronounced given the number of
users trying to use the application on mobile phones when it does not work on mobile phones.
They may not be able to “unhide” the number.
The Web Content and Usability Group identified this as a top error in their usability testing,
where a participant failed to type his SSN correctly:
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p56
● “The error message for the SSN field was too vague and didn’t help the participant. He
made several attempts to correct the error and failed.”
If you enter your SSN incorrectly in the online application, you will be routed to manual identity
verification, which is the root cause of most of the backlog.
Recommendations:
● Change UIOnline application so the default entry of SSN shows the SSN digits instead of
*** ** **** (with an option to “hide”).
○ Status: complete. This launched 9/7/20.
The security questions to create an account are unnecessarily complex.
The current account creation requires a username, password, four security questions, a security
image, and a custom passphrase key. While we appreciate the attempt at thorough security,
these measures create an unnecessary barrier.
According to the recent Voice of the Customer report, claimants who have forgotten the answers
to their security questions is a top complaint of social media and chatbot users with login issues.
Security questions, which are hard for users and present little obstacle to hackers, have been
discouraged by researchers since at least 2015.
7
Recommendations:
● Replace the online account and security questions / images / pass phrase keys with an
online account in adherence with NIST Identity Assurance Level 2 & Authorization
Assurance Level 2 (IAL2 / AAL2) (as defined in NIST special publication 800-63-3).
7
https://security.googleblog.com/2015/05/new-research-some-tough-questions-for.html
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p57
Looking Ahead to the Future
This section of the report identifies challenges related to EDD’s ability to scale to meet
constituents’ needs in a future unemployment surge, without having to maintain
pandemic-level staffing at all times.
Redirect the BSM staff back to their previous positions. Reboot the BSM
project.
It is tempting to carefully specify a brand new system which will solve all of these problems at
once. Nearly every government agency has a “next generation,” or “system 2.0” project in
procurement or development. Only 6.4% of Federal IT projects with budgets over $10M are
successful
8
. Both system requirements and available technology change faster than the
timelines for these procurements, such that they are often obsolete before they even start
EDD has a Benefit Systems Modernization (BSM) project, which is 3 years into
requirements-gathering. The contractor has not yet been selected and not a single line of
software code has been written. The rapidly changing requirements and increased workloads of
pandemic unemployment have increased the urgency of many IT changes.
We recommend immediately redirecting any staff from BSM back to their previous roles, and
once the backlog is eliminated, restarting the BSM project with a different approach. The new
approach should be a parallel iterative modernization effort based on areas of critical need.
Appendices D, F, and H have further relevant recommendations. EDD IT and ODI have already
demonstrated success in integrating ancillary systems for claims recertification and document
upload. The updated identity verification solution will be another small, quick improvement
that can be expanded into the future.
Update: CDT has informed EDD of the need to retool the BSM project.
8
https://www.brookings.edu/blog/techtank/2015/08/25/doomed-challenges-and-solutions-to-
government-it-projects/
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p58
Monitor reasons for disqualified claims.
Given the pandemic, there is an expectation that some rules (like responding with identity
verification documents within 10 days) will be relaxed and not cause for immediate
disqualification, even if it’s technically acceptable to disqualify for such a reason. Any
disqualification justification may become desirable because it removes a claim from the
backlog, but in many cases these disqualified claims will simply be escalated or appealed.
Recommendations:
● EDD can defend against perverse incentives by carefully monitoring reasons for
disqualification, in particular for areas that constituents and legislators may perceive as
more relaxed.
○ For example: how many people were disqualified for responding to an identity
verification request too late (more than 10 days)? Is the number of people serving
False Statement Weeks going up, and why? Are a high number of legislative
escalations related to False Statement Weeks, and why?
○ EDD may want to consider reaching back out to these claimants to understand
how they got stuck.
● EDD should include a report of disqualifications by reason for review in its incident
command meetings.
Monitor appeals outcomes for signs of areas to be addressed.
When claimants are disqualified, they receive a Notice of Determination (DE 1080), informing
them of the reason for the disqualification and their legal right to appeal within 30 days of the
notice.
There are two levels of appeal. The first or lower level is the appeal to an Administrative Law
Judge (ALJ). The ALJ can overturn, agree with, or modify the EDD's decision. The ALJ's decision
may be appealed to a second or higher level of appeal to the Board. The Board's decision is the
California Unemployment Insurance Appeals Board’s (CUIAB) final decision. However,
claimants and employers who disagree with CUIAB's final decision may appeal to the California
Superior Court system, which is outside of the CUIAB.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p59
EDD should keep a close eye on the percentage of appeals relative to overall claim volume, and
the percentages of decisions in favor of the claimant - either (or both) going up is indicative of
an upstream problem that could be addressed.
Currently, 49% of appeals are overturned at the lower authority, and 9% of the 51% who might
appeal get further overturned at higher authority.
Recommendations:
● Track appeals volume and percentages of decisions in favor of the claimant.
● Include decisions in favor of the claimant, sorted by Issue, in the incident command
meetings in case upstream action needs to be taken.
Clarify EDD’s purpose and incentives
An often-heard refrain from EDD staff is “unemployment is not an entitlement program, it’s an
eligibility program.” There are numerous and strict requirements to qualify, and to continue to
qualify.
To-date, EDD’s measurements and incentives have all revolved around preventing
overpayments. There is an entire area of training and a specific team dedicated to Overpayments;
there is no such team called Underpayments. There is a dedicated Fraud office that looks for
Overpayments; it does not look for lack of payments.
The U.S. Department of Labor measures California on its improper payment rate, requiring a
rate under 10%.
9
California’s EDD has met this year after year: from July 1, 2016 through June
30, 2019, its improper payment rate was 8.019%. But the federal UI Performance standards also
require that 87% of first payments be made within 21 days, and California has never met this
standard--it was 75.6% for the four-quarter period ending March 2020.
10
It is not realistic to expect a department designed to prevent overpayments to shift overnight to
focus on avoiding underpayments. We recommend EDD hold underpayments higher in priority
9
UI Performs core metrics: https://oui.doleta.gov/unemploy/pdf/Core_Measures.pdf
10
https://oui.doleta.gov/unemploy/docs/Score_Cards/2020/040119-033120.xlsx
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p60
than overpayments. This will require that California restructure and measure the Department
accordingly.
Address the coming spike in holiday determinations workload before it’s too
late
Every year, holiday determinations create a six-week backlog in the Work Share office in
January (an office that otherwise does not have a backlog). The existing backlog and the
anticipated growth in Work Share claims will make for a catastrophic new backlog in January
2021 if steps are not taken.
In Work Share, every two weeks the employer and employee collectively file Form DE 4581WS,
the Employer's Work Sharing Certification, listing the hours worked and wages paid for that
two week period. (This is effectively the recertification process for people on Work Share.)
A “holiday det” (holiday determination) is invoked when a holiday occurs during this two-week
period. It means that a claims representative must manually compare the employer’s holiday
schedule and holiday pay policy against each employee’s plan, for each Form 4581WS. Given the
number of holidays occurring between November and January each year, this caused an
estimated six week delay in processing Work Share claims in January the previous two years.
As a point of comparison, on August 18, 2020 Work Share was working July 27 claims (17
business days old). Last year, on August 18, 2019 Work Share was working same-day claims.
Extrapolating the anticipated Work Share delay in January 2021, last year’s six-week backlog
will become a 102-week backlog, made even worse by the uptick in percentage of Work Share
claims relative to last year.
We have seen no credible evidence that new hires into Work Share could be productive enough
by January to stave off this surge. The only feasible path is automation of holiday dets. This
would also prevent future weeks-long backlogs in January of low-volume years.
Recommendations:
● In collaboration with the Work Share front-line representatives, build upon the new
Work Share application to create functionality to fully automate the holiday
determination process.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p61
Plan to respond to incorrect 1099Gs in January 2021.
Recipients of unemployment payments will receive a 1099G tax form in January 2021. If a claim
was paid to the incorrect individual, that (wrong) individual will receive a 1099G that they will
want to correct with EDD.
Recommendations:
● There should be clear messaging, processes, and support planned starting now to make
this a seamless process in January.
Simplify policy and automate key processes to enable scaling up and down to
meet future demand
EDD must prepare for future surges by ensuring that the business processes that require more
manual effort relative to the number of claims are set up to scale up and down. A key learning of
this crisis is that the complexity of the policies, processes, and technology systems have made
scaling difficult because of the length of time that is needed to train new employees. As long as
the training manual is 800 pages, the manual process will never be able to scale up quickly.
As described earlier, EDD has struggled with processing incoming mail and processing identity
verifications in particular as claims volume has surged with the pandemic. These areas have
lacked automation and therefore required the highest number of additional staff hours to meet
the increased demand.
EDD can future-proof itself in these areas by moving to approaches that automatically scale
based on demand. This also prevents EDD from overspending on these tasks between surges.
Recommendations:
● Reevaluate policy and process, particularly in the areas that have been the greatest
bottlenecks, and simplify wherever possible. In this process, it will be necessary to push
back on “policy myths” and not assume that changes cannot be made unless a full
investigation is made. It is extremely common to see unhelpful policies persist because
of beliefs that they are required by state or federal regulations when in fact there is
much more flexibility.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p62
● (As above) Centralize mail/scanning, so the need to measure and scale capacity for this
function resides in one place.
○ Work with Human Resources to have a rapid staffing expansion plan in place at
all times for the centralized document processing center. In the current
pandemic, it was possible to hire hundreds of call agents but not the 40
additional employees needed to open and scan mail.
○ For the sake of evaluating all options, explore shifting to an external
mail/scanning vendor who would be paid by volume.
● (As above) Automate identity verification, which prevents the need to scale both identity
verification staff and recomputation staff in a surge.
EDD employees consider many measures temporary.
Many employees told us that certain technical automations, policies, or other improvements
are “only temporary” and “will be rolled back” after the pandemic surge. We appreciate that
some flexibilities and processes are tied to the pandemic, but also encourage EDD to consider
formalizing as many as possible, particularly around technology automations. This will prevent
the need to exercise flexibilities the next time around.
Conclusion
The challenges EDD faces in this pandemic are not unique. Across the country, departments
charged with delivering UI are struggling to adjust quickly to enormous changes in policy,
program, and volume of claims while hobbled by overly complex, outdated legacy systems.
EDD’s peers in other states also lacked the muscle to effectively update their own policies and
practices based on data and evidence about what works, and to effectively allocate resources.
The fact remains, however, that no matter how common this situation may be, it is
unacceptable. The State must deliver this benefit to those who qualify within a time frame
that’s relevant to the well-being of the claimant, and it was failing to do that for too many
Californians.
When the Strike Team started, the backlog was growing unbounded. Soon the growth will stop,
and while there is much work left to do, this year and for many years in the future, the
Department now has new tools to help them understand and manage down the backlog. Within
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p63
weeks, new claimants who apply after the new automatic identity verification solution is
operational are far less likely to see delays. Some important changes to EDD communications
and claimant interactions are underway that will reduce the confusion claimants have
experienced.
In the course of the Strike Team’s 45 day engagement with EDD, we have seen a remarkable
transformation in the department’s approach, in their capacity to use data to make critical
decisions, and in their willingness to take bold steps to address the crisis. From the beginning,
the Director’s message was that the Strike Team’s analysis and ideas were welcome and needed,
and while there was sometimes tension and differences of opinion about the appropriate course
of action, by the end of the six weeks, the level of collaboration and enthusiasm for new
methods exceeded even our highest expectations. We end our work with a corresponding
enthusiasm for this team and what’s ahead for them. We are proud of our work as a Strike
Team, but even more proud of EDD’s work as a department rising to a challenge with newfound
insights and skills.
While we wish that the backlog in claims could be eliminated overnight, we know that the plan
EDD has in place now is the best one possible under the circumstances, and that the greatest
number of people in need will get the help they need with the greatest speed. There are many
aspects of the experience of accessing unemployment insurance that still need significant
improvement before this meets or exceeds the expectations of the administration and of the
public, but we end this work with greater confidence that EDD, in partnership with ODI, CDT,
and other departments and agencies, is on the right path to make these improvements. Every
member of the Strike Team stands ready to support that path on behalf of the people of
California.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p64
Appendices
Appendix A: Data-Driven Definition & Resolution of the Backlog
Appendix A2: Backlog Data Diagram
Appendix B: Acceleration Plan for Identity Verification
Appendix C: Acceleration Plan for Legislative Case Management
Appendix D: Plan for evaluating and further iteratively improving the claimant experience
Appendix E: Acceleration Plan for Wage Verification
Appendix F: Software Development Best Practices
Appendix G: Current Claimant Experience Diagram
Appendix H: Recommended Future State Claimant Experience Diagram
Appendix I: List of Suggested Process Metrics
Appendix J: List of Suggested System Metrics
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p65
Appendix A: Data-Driven Definition & Resolution of the
Backlog
Summary
This report outlines a recommended definition of the backlog; recommends a regular
operational cadence for identifying and resolving major drivers of backlogged claims; and
explains in technical detail how to instrument a backlog report.
EDD’s overall goal should be to fully resolve 87% or more of claims within 21 days of their
submission date (within 10 days for Work Share claims).
Defining the Backlog
Number of claimants by unique ID (SSN/ECN) with an open claim associated
with:
● Any claim in UI Online that is pending staff review OR
● A work item in BCIS that could prevent determination OR
● A work item in CUBS that could prevent determination OR
● AskEDD / EDDCOM queue that would prevent determination OR
● Claims Review Sheet (messages from 8-8 call center) OR
● Legislative escalations
WITH
an earliest open UI/PUA application submission date of > 21 days prior to
today
(or > 10 days if earliest open claim is WS)
- AND -
Unopened mail by weight (short-term) / by piece (long-term), broken down
by:
● All unopened mail
● Appeals
Helpful Explanations:
●
Any application in UI Online that is pending staff review
○ This includes any application filed electronically, that was not automatically
processed, that has not yet been manually moved into a work queue.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p66
●
A work item in BCIS that could prevent determination
○ BCIS is the IT system that tracks identity verification and recomputation work
items.
○ Not all work items in BCIS pertain to unemployment, and not all work items
could prevent determinations (e.g. automated system messages). See Appendix A
for the complete list of BCIS work item queues and which ones count towards the
backlog definition.
●
A work item in CUBS that could prevent determination
○ CUBS is the IT system that tracks the unemployment claim processing work
items.
○ Not all work items in CUBS could prevent determinations (e.g. automated
system messages). See Appendix B for the complete list of CUBS work item
queues and which ones count towards the backlog definition.
●
AskEDD / EDDCOM queue that would prevent determination
○ AskEDD is the online messaging system that anyone can use (without logging
in).
○ In theory, all claimants with issues using AskEDD should be captured in other
buckets. If a claimant uses AskEDD to raise an issue that is not captured
elsewhere, that will require a review to determine why that claim did not already
have a work item attached, and to resolve all other claims experiencing that same
issue.
●
Claims Review Sheet (messages from 8-8 call center)
○ This is a central spreadsheet of messages captured from the 8a-8p cell center.
○ In theory, all claimants with issues using the call center should be captured in
other buckets. If a claimant uses the call center to raise an issue that is not
captured elsewhere, that will require a review to determine why that claim did
not already have a work item attached, and to resolve all other claims
experiencing that same issue.
●
Legislative escalations
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p67
○ This does not currently exist, but will be a central spreadsheet of all
constituent issues shared by legislators.
○ In theory, all claimants raising issues to their legislator should be captured in
other buckets. If a claimant uses their legislator to raise an issue that is not
captured elsewhere, that will require a review to determine why that claim did
not already have a work item attached, and to resolve all other claims
experiencing that same issue.
●
Unopened mail by weight
○ Weighing mail ensures it is not allowed to accrue indefinitely, artificially
lowering other work item volumes.
○ Centralizing all mail to the document processing center will allow mail to be
tracked by piece; weight is an interim measure.
Resolving the Backlog
EDD should immediately establish a semi-weekly "incident command" update between EDD
leadership and Strike Team leadership.
At each meeting, the team should review the following:
1. The three largest contributors of the backlog
○ This should focus on a recent (less than 24h old) refresh of the "backlog"
number and its contributing metrics as described above.
○ What can be done to solve / automate them?
2. Progress on implementing prior suggestions
○ EDD staff should provide brief (less than 1min each) updates on in-flight short
term suggestions, to include any blockers for senior leadership to immediately
clear.
3. The success of implemented suggestions
○ Is it working? Are those areas going down as expected?
4. Leading indicators
○ Disqualifying rates and reasons
■ Are these going up disproportionately? Where? Why?
○ Appeals rates, success rates, and reasons
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p68
■ What are the top reasons for successful appeals? What can we do to
accurately decide those claims the first time, going forward?
5. Open space for proposal for other solutions
○ Ideas that can resolve other areas of the backlog, or prevent areas of backlog.
Current Action Items:
1. Instrument the automation of this backlog dashboard, following the directions in this
report, leveraging the CDT Dashboard team.
Technical Instrumentation of the Backlog Report
This section details how to instrument a dashboard reflecting the backlog on a daily basis.
First, some important considerations:
Claimant ID
Claimants are identified either by their Social Security Number (SSN) or their ECN (EDD Client
Number).
In theory, EDD should stop using social security numbers as identifiers, but that is low priority
relative to the situation at hand, and can be solved for later.
Unique Claimants
While a claimant should only have one claim open at a time, in practice, this is not true. Many
claimants have multiple open claims. Since 4/28/20, 272,103 unique claimants have filed more
than one claim. A claimant should only appear in the backlog once, even if they have filed 131
claims (the highest number filed by one individual so far).
Therefore, this data should be unique by claimant ID (SSN/ECN), based on oldest open claim
submission date.
Closed claims do not appear in the backlog.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p69
Dates
The important date for calculating backlog is the application submitted date which is found in
UIMod. This is often different and earlier from a filing date (the date a claim is established in
SCDB). This is also different from an effective date. It’s possible to file a claim on August 30,
2020 that is effective back to February 28, 2020.
Systems
● UIMOD database (underlies both UIO and CUBS)
● UIO (UI Online)
● BCIS (Benefits Claim Inventory System)
● CUBS (California Unemployment Benefit Services)
○ Confusingly, BCIS and CUBS look exactly alike, but are maintained by separate
teams and are distinct applications/databases.
● SCDB (Single Client Database)
● BICC (Business Intelligence Continuity Center - data warehouse)
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p70
Appendix A2: Backlog Data Diagram
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p71
Appendix B: Acceleration Plan for Identity Verification
This document was written with the expectation that CDT & EDD will procure a vendor for
enhanced, automatic identity verification for integration into the beginning of the user flow
through UI online.
We further expect that vendor will be capable of identity proofing to NIST Identity Assurance
Level 2 & Authorization Assurance Level 2
11
(IAL2 / AAL2) (as defined in NIST special publication
800-63-3), and that this proofing will deprecate all other identity "verification" processes at
EDD for purposes of approving claims submitted through UIO.
We estimate below that such automation would have prevented 4,255,114 claims from having to
be manually reviewed to-date. Approximately 77.69% of claims currently being processed
manually could be automated going forward, bringing UIOnline overall automation closer to
91%, with greater fraud protection.
For the week ending August 27, 2020, this means approximately 147,280 claims of the 189,575
manual claims would have been automatically processed. This leaves approximately 6,042
manual claims per day (down from 27,082 manual claims per day).
Our experience is that these sorts of login integrations can be executed in less than 8 weeks,
given a few key constraints. Our specific recommendations for this implementation effort are:
1. Name an implementation "tiger team", comprising technical staff from the ID vendor,
the UIO vendor, and EDD IT staff, not to exceed 6 people.
2. Make that team's only full-time priority the implementation and delivery to
claimants of the new ID proofing and login.
3. Inform all other projects and teams that they are to treat any asks from the "tiger
team" as their only priority, when they happen.
4. Have this team be led, and represented to senior EDD leadership, by a single person
working for EDD (not a vendor).
11
NIST IAL Level 3, the only higher level of assurance possible, requires physical in-person
proofing, which defeats the purpose of filing online, especially in a pandemic.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p72
5. Have that representative read out daily status and blockers to EDD leadership
empowered to enforce prioritization and staffing decisions across both EDD and the
vendors.
In parallel to this technical team, for a successful implementation, EDD must also:
● Determine an alternate plan of action (other than today’s manual identity
verification) for when a new verified claimant’s SSN is already assigned to a DI claim.
● Update its policy (22 CCR § 1326-3. Identity Verification) to reflect NIST IAL2/AAL2
requirements, and eliminate irrelevant requirements. For example, once someone is
identify-proofed at IAL 2, EDD has a high level of confidence that the person is who they
say they are. “Substantiation documents” for the purpose of identity verification are no
longer required (because IAL2 includes required address verification), and an
out-of-state mailing address is now not an issue. Preferably, this updated policy will
simply point to the federal NIST standards.
● Develop a mitigation plan with the selected vendor as to what will happen if a
potential claimant fails the automated identity check.
● Develop a plan with EDD’s fraud and investigations offices to ensure they have the
data reporting they need for their work.
● Update the identity verification process done for paper claims to be consistent with
decisions made above.
● Update or eliminate Forms DE 1326C and DE 1326CD, per decisions above.
● Develop a mitigation plan for field representatives who believe they are interacting
with fraudulent claimants downstream. It must be clear that it is not their role or
responsibility to relitigate a claimant’s identity once it has been established to NIST
standards. It also needs to give them a pathway to confirm a claimant’s identity by
phone or other interactions.
We recommend this policy team be a similar “tiger team” following the format above, and that
the policy tiger team have exactly one (1) person who participates in the technical tiger team
meetings and ensures the two efforts are in sync.
UIOnline / CPT Rules
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p73
The use of an enhanced, automatic identity verification solution will make the following rules
obsolete. They should be removed/turned off from the system.
● Claimant submitted the claim application with an ECN.
● SSN not verified; Invalid SSN.
● SSN not verified; Date of Birth does not match.
● SSN not verified; Name does not match.
● SSN not verified; SSA has aka SSN.
● SSN not verified; Claimant Deceased {0}: Determine if person filed is an Authorized
Executor of the deceased and handle appropriately.
● SSN not verified; Person is a public figure.
● Claimant states that the Social Security Administration did not issue them their SSN.
● SSA interface was unavailable. Run SSA.
● Name provided by claimant did not match DMV. Verify name on DMV response to
name provided by claimant.
● DOB provided by claimant did not match DMV. IDAAR required.
● Driver License or ID not verified. Multiple DMV records found.
● Driver License or ID not verified. No DMV record found.
● DMV verification not ran. Run DMV.
● Claimant did not ID prior UI/DI Claim filed within 2 years.
● Claimant did not correctly ID prior UI/DI Claim filed within 2 years.
● SCDB interface was unavailable to verify UI, DI/PFL prior claim information.
● NO DL/ID provided
● Out state DL/ID provided
The following three rules are meant to identify instances when an imposter already filed a claim
against the provided social security number and another person is under that SSN in the
system:
● First Name, Last Name or DOB claimant provided did not match with CUBS data.
● The Date of Birth provided by Claimant did not match with SCDB.
● The name provided by Claimant did not match with SCDB.
We have proposed turning these three rules off, and replacing them with a new, more specific
rule with fewer false positives.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p74
Since the inception of UIOnline, the above rules have flagged 4,211,580 claims for manual
process. Replacing them with an automated provider can automate that workload for EDD.
Policy
The current California Code of Regulations requiring updates are:
§ 1326-2. New Claim for Unemployment Insurance Benefits -Filing and Contents.
(a) “New Claim” means an application for the establishment of a benefit year and a
computation of the maximum benefits payable and the weekly rate. (For a general
description of the claims filing procedures for unemployment benefits, see Section
1326-1 of these regulations.)
(b) Any person who is an unemployed individual, as defined in Sections 1252 or 1279.5 of
the code, may file a new claim. (See Sections 1252-1, 1252.1-1, and 1252.2-1 of these
regulations.) The claimant shall file the new claim by contacting the department and
shall set forth:
(1) His or her first and last name, and middle initial if applicable.
(2) His or her social security account number, and any other names and social security
account numbers by which the claimant is or was known.
(A) The department may require the claimant to verify the social security account
number as being the one issued to him or her by the Social Security Administration if the
information available to the department indicates that the social security account
number presented by the claimant may belong to another individual, is not a valid social
security account number, or was never issued by the Social Security Administration, or
that the wages shown in the base period of the claim may belong to another individual.
(B) If the department requires a claimant to verify the social security account number
which he or she has provided to the department during the claim application, the
claimant must submit verification of his or her social security account number through
the Social Security Administration, or he or she may submit to the department a copy of
his or her annual statement issued to him or her by the Social Security Administration.
(3) His or her date of birth, including month, day, and year.
(4) His or her gender.
(5) His or her current mailing address.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p75
(6) His or her current residence address.
(7) His or her driver's license number or identification card number, provided that the
driver's license or identification card was issued by a local, state, or federal agency, or a
foreign government, and the card contains his or her full name, date of birth, and
photograph.
(8) The date he or she last worked for his or her most recent employer.
(9) The reason he or she is no longer working for his or her most recent employer. The
department shall promptly notify the individual's most recent employer of the reason
the individual has given as to the reason he or she is no longer working for his or her
most recent employer. The department may use the claimant's statement as to the
reason he or she is no longer working for his or her most recent employer to assign the
individual an appropriate seek work plan.
(10) The name and mailing address of his or her most recent employer, except in cases
where due to reasons beyond the claimant's control, the claimant does not know the
name or address of his or her most recent employer. For example, the claimant may
have no record of wages paid, or his or her most recent employer may now be out of
business, the employer may have moved, or the employer may have died.
(11) The name or names of his or her base period employer or employers; the estimated
wages that individual earned during the base period; and the approximate periods of
employment for that individual with the base period employer or employers. If the
information available to the department indicates that the wages reported for that
individual may not belong to that individual, the department may require that individual
to provide information to substantiate that he or she earned the reported wages
(12) Whether he or she in the 19 months preceding the filing of a new claim served in the
United States armed forces, or worked for an agency of the State of California, of another
state or of the federal government, or worked for an employer in another state.
(13) Whether he or she has filed a claim for unemployment insurance or disability
insurance against California, against another state or against the federal government in
the past 24 months.
(14) Whether he or she is a member of a union, and if so, the name and number of the
union local, and whether he or she is registered as out of work with his or her union.
(15) Whether he or she is unemployed or working part time and knows the law requires
true and complete answers, and that he or she may be required to register for work (as
defined in Section 1251-1 of these regulations).
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p76
(16) A statement that the claimant is a citizen or national of the United States and, if not,
that the claimant is in a satisfactory immigration status, as defined in Section
1326-13(c)(1) of these regulations. The claimant shall also make the statement set forth
in Section 1264-1(d) of these regulations, as to his or her citizenship or immigration
status during the base period.
(17) Such other information as the department may require.
§ 1326-3. Identity Verification.
(a) A claimant will be required to provide information to the department as described in
Sections 1326-2, 1326-4, 1326-5, or 1326-6 of this Division. The information provided
by the claimant must be sufficient for the department to:
Appendix B: Acceleration Plan for Claimant Identity Verification - p6
(1) establish the identity of the claimant, and;
(2) verify that the wages reported under the social security number provided belong to
the claimant.
(b) If the information provided to the department by the claimant under Sections
1326-2, 1326-4, 1326-5, or 1326-6 of this Division does not sufficiently establish the
identity of the claimant, or if the department cannot verify that the wages reported
under the social security number provided belong to the claimant, the department shall:
(1) Require the claimant to verify his or her identity by presenting a photo identification
and one of the following document(s) as defined in Section 1251-1 of this Division:
(A) Social security number verification.
(B) Date of birth verification.
(C) Address verification.
(D) Employment data.
(2) Ask the last employer of the claimant, and all of the claimant's base period
employers, to provide the following information:
(A) Any other names used by the individual who earned the wages.
(B) Any other social security numbers used by the individual who earned the wages.
(C) Dates of employment for the individual who earned the wages.
(D) Last known phone number for the individual who earned the wages.
(E) Last known mailing and residence addresses of the individual who earned the wages.
(c) If the information available to the department indicates that the identity of the
claimant may not be the same as the individual who earned the wages reported to the
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p77
department, the department shall request the individual who claims to be the true
owner of the identity to certify under penalty of perjury whether or not he or she filed
the claim for unemployment insurance compensation.
(d) The claimant shall have a reasonable opportunity, as determined by the department,
to provide the information requested pursuant to paragraph (b)(1) above to the
department. The claimant shall have the right to request additional time to provide the
requested information to the department.
(e) If, within ten (10) days from the mailing date of the request, the claimant has not
supplied the requested information, and has not contacted the department to request
additional time to provide the requested information, the department shall grant or
deny the benefits based on the information available to the department, pursuant to
Section 1253(a) of this Division.
Note: Authority cited: Sections 305 and 306, Unemployment Insurance Code. Reference:
Sections 1085, 1092, 1257, 1326 and 1327, Unemployment Insurance Code.
We would instead suggest:
§ 1326-2. New Claim for Unemployment Insurance Benefits -Filing and Contents.
(a) “New Claim” means an application for the establishment of a benefit year and a
computation of the maximum benefits payable and the weekly rate. (For a general
description of the claims filing procedures for unemployment benefits, see Section
1326-1 of these regulations.)
(b) Any person who is an unemployed individual, as defined in Sections 1252 or 1279.5 of
the code, may file a new claim. (See Sections 1252-1, 1252.1-1, and 1252.2-1 of these
regulations.) The claimant shall file the new claim by contacting the department and
shall set forth:
(1) His or her first and last name, and middle initial if applicable.
(2) His or her social security account number, and any other names and social security
account numbers by which the claimant is or was known.
(A) The department may require the claimant to verify the social security account
number as being the one issued to him or her by the Social Security Administration if the
information available to the department indicates that the social security account
number presented by the claimant may belong to another individual, is not a valid social
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p78
security account number, or was never issued by the Social Security Administration, or
that the wages shown in the base period of the claim may belong to another individual.
(B) If the department requires a claimant to verify the social security account number
which he or she has provided to the department during the claim application, the
claimant must submit verification of his or her social security account number through
the Social Security Administration, or he or she may submit to the department a copy of
his or her annual statement issued to him or her by the Social Security Administration.
(3) His or her date of birth, including month, day, and year.
(4) His or her gender.
(5) His or her current mailing address.
(6) His or her current residence address.
(7) His or her driver's license number or identification card number, provided that the
driver's license or identification card was issued by a local, state, or federal agency, or a
foreign government, and the card contains his or her full name, date of birth, and
photograph.
(8) The date he or she last worked for his or her most recent employer.
Appendix B: Acceleration Plan for Claimant Identity Verification - p8
(9) The reason he or she is no longer working for his or her most recent employer. The
department shall promptly notify the individual's most recent employer of the reason
the individual has given as to the reason he or she is no longer working for his or her
most recent employer. The department may use the claimant's statement as to the
reason he or she is no longer working for his or her most recent employer to assign the
individual an appropriate seek work plan.
(10) The name and mailing address of his or her most recent employer, except in cases
where due to reasons beyond the claimant's control, the claimant does not know the
name or address of his or her most recent employer. For example, the claimant may
have no record of wages paid, or his or her most recent employer may now be out of
business, the employer may have moved, or the employer may have died.
(11) The name or names of his or her base period employer or employers; the estimated
wages that individual earned during the base period; and the approximate periods of
employment for that individual with the base period employer or employers. If the
information available to the department indicates that the wages reported for that
individual may not belong to that individual, the department may require that individual
to provide information to substantiate that he or she earned the reported wages
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p79
(12) Whether he or she in the 19 months preceding the filing of a new claim served in the
United States armed forces, or worked for an agency of the State of California, of another
state or of the federal government, or worked for an employer in another state.
(13) Whether he or she has filed a claim for unemployment insurance or disability
insurance against California, against another state or against the federal government in
the past 24 months.
(14) Whether he or she is a member of a union, and if so, the name and number of the
union local, and whether he or she is registered as out of work with his or her union.
(15) Whether he or she is unemployed or working part time and knows the law requires
true and complete answers, and that he or she may be required to register for work (as
defined in Section 1251-1 of these regulations).
(16) A statement that the claimant is a citizen or national of the United States and, if not,
that the claimant is in a satisfactory immigration status, as defined in Section
1326-13(c)(1) of these regulations. The claimant shall also make the statement set forth
in Section 1264-1(d) of these regulations, as to his or her citizenship or immigration
status during the base period.
(17) Such other information as the department may require.
Appendix B: Acceleration Plan for Claimant Identity Verification - p9
§ 1326-3. Identity Verification.
(a) A claimant will be required to provide information to the department in accordance
with the most recent federal NIST Special Publication 800-63 standards
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p80
Appendix C: Acceleration Plan for Legislative Case
Management
Overview
In the creation of the backlog definition, all stakeholders have agreed that resolving constituent
complaints raised to their legislators is a high priority for the Department. When we met with
the EDD Legislative Office to incorporate their data into our dashboard instrumentation, we
learned they have no tracking tools at all -- individual constituent concerns are handled
individually by email.
This means there is no data to incorporate into the dashboard, but also that some constituent
concerns are likely going unresolved -- and also that a lot of constituent personal information
is being transmitted by email.
Before the pandemic, EDD’s Legislative Office received approximately 1,000 unemployment
insurance related inquiries each year. Since the beginning of the pandemic, they have received
more than 1,000 cases per day, with a current backlog estimated at 18,000 unprocessed
requests.
This acceleration plan is for the development of a legislative escalation request tracker for EDD.
This tracker will serve the EDD Legislative Office for the long-term, including for legislative
escalations for other constituent requests such as Paid Family Leave and Disability Insurance.
Project Schedule
August 27, 2020: Kick-off meeting
September 4, 2020: Initial Minimum Viable Product (MVP) version live for testing by the EDD
Legislative Office by
September 11, 2020: Expected project completion
September 11-25, 2020: One sprint of iterations/improvements based on the feedback from the
EDD Legislative Office
Project Team
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p81
The existing OSaaS Salesforce contracting team from Work Share is being leveraged for this
project, in collaboration with the EDD Legislative Office. EDD CIO Rita Gass is the EDD IT
Executive Sponsor.
Scope of Work
Overview:
Give the EDD Legislative Office the ability to centally track legislative escalation requests
related to unemployment claims.
Proposed Path:
Use standard Account object to represent individual legislative offices, by legislative member
name.
Use standard Contact object with two custom Record Types: Constituent and Legislative Staff.
Hide all tabs that are not Account, Contact, or Reports
For Contact[Constituent], the following standard + custom fields:
● Account = the legislative office they called into
● Name
● Email
● Phone
● SSN (custom, string)
● Obscured ID (custom, read only, formula field that shows last 4 of SSN)
● Category of Issue (custom, picklist)
○ Match this to the AskEDD dropdown as a start
○ Train EDD legislative office to be able to update this picklist
● Status (custom, picklist)
○ Open
○ In Progress
○ Resolved
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p82
○ Re-Opened
● Status Last Updated Date (custom, read-only, auto-completed by rule when Status
picklist ISCHANGED())
● Assigned Staff (custom, optional lookup to Legislative Staff on their assigned
Account)
● Description of Issue (custom, text area)
● Include Activity History
○ From here, an EDD user can send an email back to the legislative staff member
with an update
● Include Field History on Status picklist
● Rename “Log a Call” button to “Log an Activity”
● Hide all other fields and related lists for simplicity of user interface
For Contact[Legislative Staff], the following standard + custom fields:
● Account = the legislative office they work for
● Name
● Email
● Phone
● Include Activity History
● Include custom Related List for Lookup: Assigned Staff
● Hide all other fields and related lists for simplicity of user interface
Reports
1. The Backlog Dashboard team requires this report:
○ All Contacts where Contact Record Type = Claimant && Status != Resolved
○ Fields: Name, SSN, Status, Record Created Date
2. The initial report for each legislative office can be:
○ All Contacts where Contact Record Type = Claimant
○ All Contacts for their Account
○ Fields: Account, Name, Obscured ID, Category of Issue, Description of Issue,
Status, Status Last Updated Date
3. For the EDD Legislative Office:
○ All Contacts where Contact Record Type = Claimant
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p83
○ Fields: Account, Name, Obscured ID, Category of Issue, Description of Issue,
Status, Status Last Updated Date
○ Save a specific report of Status = “In Progress” and Status Last Updated Date >
7 days ago.
Users:
● Give each legislative office one user (80). Restrict that user to only access one account
(theirs).
● Each EDD Legislative Office staffer (6) would have their own user.
How will legislative staff use this tool?
● They can log in and:
○ Create new Contacts against their Account for constituent requests, avoiding
the need to transmit constituent PII over email.
○ Check the status of existing constituent requests (by searching their contacts
or looking at a report).
○ Review report #2 referenced above (filtered to just their account automatically
by Salesforce filters) of all pending constituent requests.
How will the EDD Legislative Office use this tool?
● They can log in and:
○ Send email updates on a Contact to the Assigned Staff
○ See report #3 referenced above of new open requests
○ Update the status of requests
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p84
Appendix D: Evaluating and Further Iteratively Improving
the Claimant Experience
In addition to the incident command meetings to work down the backlog in a data-driven
manner, EDD should also have a regular cadence of data-driven improvements to the claimant
experience.
An improved claimant experience could have prevented significant churn during the pandemic,
getting ahead of confusing certification questions and having plain language claim status
readily available. Regularly working to improve the claimant experience will position EDD that
much more effectively in the next surge in unemployment claims: reducing incoming calls,
claimant mistakes, and earning public trust.
Data-Driven Drivers
Major drivers of the claimant experience include (but are not limited to):
● Google Analytics events on UIOnline
○ Top transactions
■ What are people doing the most? Filing a new claim? Checking the
status of their claim? Checking delivery of their debit card?
○ Referral sources (where are users coming from?)
○ Bounce rates
○ Drop-offs at account creation
■ This would explicitly not include people who cannot pass IAL2 identity
verification
○ Drop-offs during the application process
■ At what field/page are people dropping off the most? Why?
○ Clicks on “help” icons
■ On what field/pages do people require the most help?
○ Error pages
■ Broken links should be resolved within one week
■ What are the top errors received? Wh y? Can they be resolved? If not ,
Is t h e error message clear on what t o do next?
○ Language needs
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p85
■ Whether or not users use the "Google Website Translator" on
form s.edd.ca .gov/ form s
● Percentage of initial claims received on paper and via telephone on e
○ What are the barriers t o applying online?
● Messages received from [potential] claimants
○ Messages through UIOnline (“web correspondence” )
■ By category
○ Messages through the public website (“AskEDD” )
■ By category
○ Calls received through the call center (s)
○ Comm en ts made on social media channels (from Voice of the Customer
reports)
● Call center select ion s
○ What skills (“press 2 for XYZ” ) are callers selecting the most? Are these
also fir st?
○ How long (in seconds) does it take to complete the top 3- 5 transactions
via automated ph on e system ?
● Usability testing results
○ Top 3- 5 issues identified by WCUG
● Accessibility test in g result s
○ An y WCAG 2.0 accessibility test in g failures must be remediated in the
next spr in t .
○ Top 3- 5 issues identified by DAT
Reviewing Claimant Experience Data
1. On a weekly basis, review the data elements identified above in a m eet in g th at
includes top EDD leadership, a WCUG representative, a DAT representative, an d
a representative with the ability to directly make changes to each of the
aforementioned systems. (In other words, this person must be close enough to
the system t o know exactly how t o make changes - - not translate the discussion
to another team member .)
2. Over t im e, combine these data elements in to a dashboard for easy and regular
review. Almost all of these data elements should be available in real- time,
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p86
particularly those from Google Analytics.
3. Review im provem en t s m ade in recent meetings. What change did you expect t o
see from the change (e.g. users no longer hitting an error page)? Did the change
you expected happen ?
a . If yes: mark the item done
b. If n o: brainstorm additional solutions
4 . Make a plan for 3- 5 new improvements, prioritized based on t h e data above.
a . If the planned action will take more than one week t o implement , track it s
progress in this meeting.
5. Repeat weekly.
Categories of Inquiry
On at least a quarterly basis, EDD should review the categories available for claimants
or call center agents to categorize messages. These options should be consistent across
all mediums (AskEDD, UIOnline messages, call center).
Review categories such as “other” for emerging themes that may require their own
category. Consider removing current categories that are rarely used.
Assign a Product Owner
A single individual a t EDD should be designated a s the full- time product owner . This
person is responsible for:
● Reviewing all the data above on a regular basis (more often than the weekly
meetings)
● In t imat ely understanding the claim an t experience, including all underlying data
systems and data structures
● Prioritizing improvements to the claimant experience, weighing the benefit t o
claimants against t h e cost in time and development resources
● Participating in usability and accessibility session s
● Staying on top of what other st a t es are doing for their unemployment programs
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p87
● Coordinating changes to the claim an t experience with leadership based on
changing legislative requirements
● Advocating for policy changes that will further improve t h e claimant experience
This person cannot have another job a t EDD; they must be the product owner full- time.
Proactively Improving the Claimant Experience
The best way to ensure a positive claimant experience is to test any planned changes or
new features/functionality with actual users, ahead of launch, and to include usability,
accessibility, and actual users in the development of new functionality and features from
day one .
EDD can achieve this by:
● Creating a version of its testing environment that is accessible outside of the EDD
network
● Recruiting and supporting a group of end user testers who do not work for EDD.
These would not be employees, but would join for (compensate d) brief usability
sessions.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p88
Appendix E: Acceleration Plan for Wage Verification
Executive Summary
Layer Aleph was asked to specifically focus on opportunities for further automation and
improvement around the wage verification process.
Overall, claims representatives reported easy automation of wage verifications for Californians
with “traditional” (W2) employment, via the Franchise Tax Board (FTB). Representatives
reported infrequently having to intervene and perform wage verification for these claimants.
Challenges for wage verification arise for claimants with out-of-stage wages from the last 18
months, federal employees, and newly-separated military. These special claims require
claimants to submit specific supplementary documentation, which they cannot do online, and
thus they must file on paper. Document upload will significantly help these use cases, as they
will all be able to file electronically once document upload is live.
Other opportunities for acceleration of wage verification improvements we explored include
greater automation around Alternate Base Period wages; wage verification for PUA claimants
seeking an increase in their base rate; and claimants with past disqualifications.
Wage Verification for Claimants with Out-of-State Wages
Individuals with out-of-state wages in the last 18 months cannot file electronically on UIOnline.
Depending on the circumstances, this can be an “interstate claim” or a “combined wages
claim.” After indicating they have out-of-state wages from the last 18 months, online
applicants are directed to fax in their application:
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p89
In June 2020, 70,000 (4%) of online filers indicated they had out-of-state wages and were
redirected to fax their application. In July 2020, this number rose to 76,000.
While some wage verification can be resolved through inter-state data systems, proof of wages
from out of state from the last two years often must be provided by claimants. Since there is no
document upload, claimants must fax/mail in this proof, and cannot file online.
Allowing claimants with out-of-state wages to file online and upload their wage verification
document would eliminate this manual volume of claims and allow 4% more claimants to file
electronically, benefitting from the additional automation and efficiency that provides.
Recommendations:
● Expand document upload to include non-identity documents in the release after the
September 8, 2020 release.
● Once claimants can upload wage verification documents, change the question “Did
you work in another state and/or Canada during the last 18 months?” to not stop a
claimant from filing on UIOnline.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p90
● If there is a delay in expanding document upload for all documents, EDD should
update the verbiage on the UIOnline website to inform claimants they cannot file
electronically if they have out-of-state wages from the last 18 months, before they
begin the electronic application.
Wage Verification for Military/Veterans
Newly separated military (individuals who are now veterans) are eligible for unemployment for
more reasons than other citizens, due to federal law. These veterans cannot file for
unemployment online in California. There are currently 11,604 pending Veteran claims in CUBS.
They must provide EDD with the specifics of their federal employment on paper via Form DE
1101IAD (including SUPPLEMENTAL FORM FOR EX-SERVICEMEMBERS – ATTACHMENT A).
The only difference in the process is this ATTACHMENT A.
EDD uses the veteran’s Form DD214 as the final and conclusive proof in terms of eligibility
(based on character of discharge - you must have an honorable character of discharge) and
monetary award (based on last pay grade). This information is found on Form DD214. Form DE
1101IAD SUPPLEMENTAL FORM FOR EX-SERVICEMEMBERS – ATTACHMENT A allows a
veteran to submit their DD-214 in lieu of ATTACHMENT A, but if they do not submit it, EDD will
call them and ask them to mail it in. The form does not make this clear.
EDD needs the following fields off a DD-214:
● Character of discharge
● Record of service
● Pay grade
These fields are all available on a paper DD-214, and also available to verify electronically,
automatically, from the VA on VA.gov.
In the short-term, EDD can reduce the overall amount of manual work associated with wage
verification for veterans by:
● As soon as Document Upload for all documents is available, allow Veterans to file
electronically. This would be the normal UIOnline form, plus a requirement to upload
their DD-214.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p91
● Replace SUPPLEMENTAL FORM FOR EX-SERVICEMEMBERS – ATTACHMENT A with
a requirement to attach the DD-214.
In the long-term, EDD can fully automate these claims:
● Allow Veterans to file on UIOnline as normal
● Use the Veteran Verification API (under service_history) from VA.gov, integrated into
the UIOnline flow, to verify character of discharge, pay grade, and record of service
directly from the VA.
○ The fields needed are discharge_status, pay_grade, and deployments.
○ Since UIOnline will use ID.me, the same identity provider as the VA, explore
any opportunities to leverage existing code/connections to access the Veteran
Verification API.
○ We recommend obtaining a level of effort from ID.me early on, but this
automation can be prioritized for later. The most important part is enabling the
separating military to file electronically and upload their DD-214.
Wage Verification for Federal Employees
Former federal government employees cannot file for unemployment online in California. They
must provide EDD with the specifics of their federal employment on paper via Form DE 1101IBD
(including SUPPLEMENTAL FORM FOR FEDERAL EMPLOYEES – ATTACHMENT B). The only
difference is this ATTACHMENT B.
This form asks for the exact three-digit Federal Identification Code (FIC) of their former federal
duty station, which field representatives reported was very difficult for people to do accurately,
and usually involves a back and forth of phone calls, despite this information being clearly
printed on the SF-8 form every federal employee is issued at separation.
Recommendations:
● Replace SUPPLEMENTAL FORM FOR FEDERAL EMPLOYEES – ATTACHMENT B with a
request for an SF-8 or SF-50. Federal employees can use the normal electronic UIOnline
form with the addition of instructions to upload their SF-8 or most recent SF-50.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p92
Wage Verification to Increase PUA Base Wages
Applicants for Pandemic Unemployment Assistance (PUA) can qualify for increased dollar
amounts based on their 2019 California income tax return data. EDD met with the Franchise Tax
Board (FTB) on 8/3/20 to begin the approval process for automatically retrieving this data for
claimants. (Given the pandemic-related tax filing grace periods, not all 2019 California income
tax returns have been filed.)
Given that EDD’s newest leadership includes individuals coming from the FTB, we are confident
that this partnership will be supported and shepherded.
Recommendations:
● Establish a permanent data sharing agreement with FTB for prior-year tax returns,
both to retroactively check for fraudulent overpayments, and to make it easier to access
this data in case of a future surge or unexpected federal program.
Alternate Base Period Wages
If a claimant cannot qualify for Unemployment based on the Standard Base Period, they may
qualify under an alternate lens: the Alternate Base Period (ABP).
From EDD’s Fact Sheet:
The “Standard” Base Period uses the wages earned in the first four of the last five
completed calendar quarters prior to the beginning date of the UI claim. The “Alternate”
Base Period uses the wages earned in the four most recently completed calendar
quarters.
If the claimant did not work during the standard 12-month base period but worked in
the most recent quarter at the time of filing a claim, they may be qualified for an
alternate base period claim. This type of claim requires the EDD to request the work and
wages history information from the claimant and the employer to determine if the
individual meets monetary eligibility requirements.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p93
Prior to the pandemic, if a claim came in as “IE/NE” (Insufficient Earnings / No Wages), they
would be manually assessed by staff who ran the Wage Tool.
During the pandemic, the macro team added automation to assess some of these IE/NE claims
against the ABP. CPT Code 49 assesses opportunities to use the Alternate Base Period.
Recommendations:
● Explore additional automation opportunities for the Wage Tool, led by the Macro
Team.
Claimants with Past Disqualifications
If a claimant has a previous disqualified unemployment claim, their new claim will be routed for
manual review. A claim could have been disqualified for many reasons, such as not earning
enough wages - a disqualified claim is not necessarily a negative flag.
Multiple supervisors raised this as an opportunity for automation, pointing out that if the
previous claim was disqualified because of insufficient wages, it is straightforward to
recalculate their eligibility based on more recent wage history. (This can be, but is not always,
separate from Alternative Base Periods.)
Recommendations:
● Draft criteria for automatically processing new claims from claimants with certain
types of previous disqualified claims.
Non-Responsive Employers
If an employer fails to respond to a Request for Wages DE 1919 form request (a request to verify
a claimant’s employment/wages) within 10 days, and the individual returns the Affidavit of
Wages DE 23A along with tangible proof of wages earned, California law requires EDD to accept
the wage information provided by the individual.
We heard many reports of specific employers consistently never responding to these DE 1919
requests. We observed that when some employers consistently fail to respond, a Wage Notice is
posted on the EDD UI Branch SharePoint site, but this is not incorporated in the automated
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p94
system. Therefore, when an employer with a large number of alleged employees fails to
respond, many claimants are left to wait 10 additional days for EDD to process their claim.
Recommendations:
● EDD should establish a formal policy that if a particular employer fails to respond to a
certain number of requests in a certain period of time, EDD will automatically accept DE
23As from their [alleged] employees for some period of time. We leave the exact time
periods for EDD to determine based on their data.
● Add automation to the system to automatically flag non-responsive employers and
allow claims representatives to skip the notice/10-day waiting period if the claimant has
established a DE 23A.
Other Automation Opportunities
Once a claimant’s identity is verified, UIOnline shows a claimant the base period wages known
to the system, and the claimant has to confirm this number. If they say no, the claim is routed
for manual review. This is known as a wage investigation.
Historically:
● 21.98% of claims routed for manual review were because the claimant indicated they
had a missing employer (CPT Code 38). This requires manual recomputation.
● 10.22% of claims routed for manual review were because the claimant indicated they
did not work for a base period employer that was returned from the base wage data (CPT
Code 40). This requires manual recomputation.
● 7.22% of claims routed for manual review were because the claimant reported they
had additional base period employers not shown in UIOnline (CPT Code 44). This
requires manual recomputation.
● 5.57% of claims routed for manual review were because the claimant reported they
had Other pay (CPT Code 36). This requires manual recomputation.
Recommendations:
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p95
● EDD should perform a data analysis of claims with CPT Code 36, 38, 40, or 44, relative
to the data returned about base period wages, to determine what data sources are
incorrect, outdated, or missing, leading to the need for manual review.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p96
Appendix F: Software Development Best Practices
We recommend that every development team implement the same baseline development
practices, to the extent possible.
Automated promotion through test stages
Minimizing manual acceptance testing should be an ongoing program goal, and the amount of
wall-clock time required for manual testing should be a key performance indicator -- to be
managed down.
The best-in-class behavior for any continuous delivery on UIO projects will be automated
promotion towards production based on tests.
Performance measurement
To optimize for agility and improvement, speed of delivery should be a top-level program key
performance indicator. Speed of delivery should be measured as the wall clock time between
having a final requirement written for a change, and that change being live for claimants.
Revision control configuration
Every development code repository should have the following settings and policies, to the
extent possible:
● Protected branch on the main development branches
● Required approval for merges or pull requests, to include code reviews
● Required quality and testing automation:
○ Automated code quality assessments triggered by code submission
○ Automated build and unit test runs triggered by code submission
○ Automated test coverage assessment triggered by code submission
● Automated rollback/rejection of changes that fail these automated checks
Once automated guardrails are in place, remove and deprecate all other code review/submission
guidelines, policy, and checklists.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p97
Appendix G: Current Claimant Experience Diagram
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p98
Appendix H: Recommended Future State Claimant
Experience Diagram
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p99
Appendix I: List of Suggested Process Metrics
We recommend that EDD leadership regularly review the following process metrics. Ultimately,
these metrics should be accessible via a central data dashboard.
The Backlog
These metrics are covered in Deliverable 5.2: Plan for Instrumenting Dashboard
Claims:
● # unique claimants with an open determining work item on a claim that was
submitted more than 21 days ago
Mail:
● Short-term: daily weights of unopened mail, by office
● Long-term: daily count of mail, by type (e.g. PO Box #), from the central mail
processing center
Overall Process Metrics
Aging reports on the following open items:
● Throttled in CPT
● BCIS (by Recomp Type & Work Queue)
● CUBS (by Work Queue)
● AskEDD (by category)
● Legislative Escalations (by category & status)
Disqualified claims
● As a percentage of overall volume
● By reason
Appeals
● Volume
● Percentage of appealed claims relative to volume of claims
● Percentage of appeals overturned at each level, by Issue
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p100
Productivity
● Daily Productivity Sheet tally (DPS) [existing]
● Estimated time to complete work based on current staffing levels and last month’s
Minutes Per Unit
○ For the backlog
○ For all pending open items
Usability
● Top 3 issues from weekly Voice of the Customer report
○ Broken down by: AskEDD, Social Media, and Call Center
● Top 5 issues identified by WCUG
● Top 5 issues identified by Digital Accessibility Team (DAT)
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p101
Appendix J: List of Suggested System Metrics
We recommend that EDD leadership regularly review the following system metrics. Ultimately,
these metrics should be accessible via a central data dashboard.
System Monitoring
Service Level Agreements for:
● CPT
● CUBS
● CUBS sync performance
● UIOnline
● BCIS
● SCDB
● ID.me
● Docusign
Usability of UIOnline
Review Google Analytics for (at least) the following metrics:
● Top page views
● Top help icon click reasons
● % mobile users
○ Overall
○ For initial UI claim filing
○ For document upload
○ For recertifying
● Abandon rates for:
○ Creating an account
○ Verifying identity
○ Completing initial UI claim
○ Uploading a document
○ Recertifying
● Time to complete the transaction for:
○ Completing initial UI claim
○ Uploading a document
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p102
○ Recertifying
● Retroactive certification tool:
○ Login responses
○ Partial saves
○ Completed certifications
○ Logins from users who completed their return already
○ Back button clicks
○ Print confirmations
○ Session timeouts
● Whether or not users click "register or login" from the UI guide Nava developed
● Whether or not users use the "Google Website Translator" on forms.edd.ca.gov/forms
The top 10 transactions should also, verifiably, not be more than two clicks away after logging
in.
STRIKE TEAM FINAL DETAILED ASSESSMENT AND RECOMMENDATIONS - p103
