CEO and Co-Founder David Yang shares a behind the scenes look at coding bootcamp job outcomes, specifically outcomes that are reported through the Council on Integrity In Results Reporting, the leading standards organization for technology bootcamps.
Hi, I'm David Yang, CEO of Fullstack Academy. And today I want to talk about bootcamp outcomes and resulting in how do I read this? Your friendly CIRR report.
Ways to Evaluate Bootcamps
So really a lot of students are wondering when they apply into a bootcamp, how do I know if a school is good or not, right? Because it can be very confusing looking at school websites. I know a lot about our website, I know a lot about our competitors' websites. A lot of them are saying very compelling things, but how do you compare them against one another.
I've always thought the value proposition of bootcamps is very high, but students want to know, how do I choose the best one? Or how do I choose the best one for me? It's a great question. So really there's a bunch of ways to evaluate bootcamps.
The general answers are you should read reviews online. You should talk to alums. I think this is probably the one of the most under utilized ones. You should visit. Most bootcamps, I know we do at Fullstack, we do info sessions both online, on the campus. Walk around and take a look. See if people look happy. See if people look like they're learning a lot. And you should, in terms of reviews, check out Course Report, SwitchUp. That's where most of our alums are posting their reviews.
CIRR Outcomes Reports
And finally, you should be looking at the outcomes report that most bootcamps now publish outcomes report of some type. This is the one published according to a group called CIRR. The Council on Integrity in Results Reporting.
Now, quick disclaimer, I'm on the board of CIRR. So that means I meet on a bi-weekly basis with the rest of the members of CIRR talk about standards, how they're working, if they're working, who's in compliance, who's not in compliance. And so, I know a lot about CIRR, I know a lot about the history of CIRR.
And now I want to share some of that with you because often times standards, and standards bodies can be kind of confusing. So I'd love to start with how CIRR started or what why it was important to start CIRR? So really, we came out in 2013, 2014, a lot of the bootcamps were really growing, very quickly. And it became confusing to compare bootcamps for students.
We met as an industry and said what we don't want to do is poison the idea of bootcamps to students, because it seems like these things are too good to be true. I think often times students are very smart. They know how to evaluate and look at a message a school is saying, and we wanted to make sure that all the best players, who are doing good work, are communicating that fairly to students. And, so, we met as a group and decided, how do we make sure that we're making the most clear and concise way that students can understand what is actually happening to alums in bootcamps.
We want to make sure that the true message was getting across. And there was actually a lot of different standards that were popping up. There's the old joke, the nice thing about standards is that there's so many to choose from. I remember that we had our standard, several competitors were putting out standards, there were several attempts to create a standards body, and out of that frustration there was a meeting to create this group called CIRR, the Council on Integrity in Results Reporting.
And what CIRR really was focused on is how to communicate this clear to students, and how do we make sure that there's truth in advertising and of what the bootcamps are saying. So, what are the key metrics of how outcomes are determined and how we think about outcomes? So, really, CIRR's goal is to create a standard that is easy-to-understand, something that students can take action on, it's transparent, and transparency is one of those things that was really at the heart of CIRR.
And really, it was clear, we had this goal of can we describe outcomes on a single page. And that they were verified. So, CIRR reports, as a member of CIRR, you have to send your report, you have to send your entire student list to an actual, real-life auditor, and what they'll do is they'll call each student, and verify that the information the school is saying matches the information that the student is reporting.
Key Metrics of a CIRR Report
So, the key metrics in this, and how it works. One is, so, this is a Fullstack Academy's report. I don't want to tell you about individual numbers because they're all available online, on CIRR.org, but how do you read this?
First thing at the top, it'll either say H1 or H2 and the year. So, H1 means students that graduated in the first half of 2018. And what happens is that this report, H1 2018, was published March of 2019. Which, you might be saying wow, that's a year apart. And the reason that works, is that all the students who graduated in the first half of 2018, they had 180 days, so about six months, to do their job search. And then, at the end of that six months, so at the beginning of 2019, is when we start doing employment verification. We start reaching out to students, how did your job search go? Of course we're in contact with them the whole time.
Then it takes us some time to compile that data, to verify that data. And, so, another one to two months to take all that information and compile that in this report. So, there is a little bit of a time lag, because it takes time to find a job. So, as you can see, there's school. It'll tell you the campus location. So, each campus is reported separately. So, for us, we report New York, we report Chicago, we report our Grace Hopper programs. We report them all separately. This is for students from January to June 30th of 2018. And it says published course length. So, how long was the course? All right, so ours is 109 days. That's about three months.
And then how many graduates are included in the report? So, here we have 117. And then it goes through all the graduation requirements. So, they had to complete the course modules, they had to complete a final project, they had to meet with our counselor, and they have to attend two career events.
And then here's where it gets you all the juicy information. Graduation data. How many students graduated within 100% of the published program length? And here, we have 100%. So these are students who graduate on time in the program. Right, so, our program is 109 days, this is students who started and finished.
And then, there's how many students graduate within 150% of the published program length? Here's something that very common in the bootcamp world. And it's actually a similar statistic in the higher education world, there's 150% mark. So, if you think about your college, most people graduate college in four years. And let's say 60%, and then there's a bunch of stragglers who finish in five and six years. The same thing happens in bootcamps, where a lot of people will take the published time, but there's some who either do, what we call a replay, they might take some time off, they might take a medical leave of absence, and then they'll come back and finish it in 150% of the time. So, this is really capturing how many students are taking a little bit longer than normal to finish the program.
And then you have two columns here. One that captures information at 90 days, and one that captures information at 180 days. This is from the time the student graduates.
So, the first section is how many people are employed in-field. In-field meaning students who are working in the technology field, or a field related to what they're graduating in. The second section is employed out-of-field. So, this is hey, I went to web development bootcamp, but now I'm working in retail, or now I'm working in data, or I'm working in hardware. So, something that's not related to what they learned in the bootcamp.
The third section is not employed. So, these people who, most simply think of, didn't get success from the program, right? They are either still looking for and seeking a job or not seeking a job. And the not seeking a job is a little complicated because if you go back to school, that also counts as not seeking a job.
And then four is non-reporting. So, non-reporting is a section where these are students who just don't wanna get in touch with you, right? And it's not that they're mad at you or that they hate you, they're just busy, might be out of the country, might just say you know what? I enjoyed my time at Fullstack, but I don't want spend more time reporting all this information. So, there's a section called for non-reporting. And non-reporting means you've made attempts, as a school, to contact the student, and they haven't gotten back to you.
When you're employed in-field, if you go to a web development bootcamp and you're employed as a web developer, there's different kinds of employment. So, most of us think I'm gonna get a full-time employed job. That means I'm working 30 or more hours. That's what I would kinda raise most as the typical success metric, right? People who, they went to bootcamp, they got a full-time job.
B, section one B, is they have an apprenticeship, internship, or contract position. So this is also, this is not bad. This is, often times people will take apprenticeships because that's just what companies do. Spotify, Google, often times, will do a contract to begin with. Spotify has a Fellowship program that's very popular.
Section one C is that they're hired by the school in-field. So, we often times, we are also a technology company. We also develop software. We also have teaching requirements. So, what we want to hire people who are really good at software development, people who really care about education. One of the best pools is students that graduate from Fullstack. A lot of times we'll hire our own students, in-field.
Section D is those who start a new company or venture. So, this is the entrepreneur groups, the people who came to learn programming, and I think this is an amazing idea. I've started several companies. If you can be the builder of your company and the leader of your company, you're far ahead of so many startups.
And E is short-term contract or freelance. So, some of our students, I think ideally, what I imagine here, is a student who wants to travel, they take some freelance work. Or sometimes, there are just companies who are looking for beginning talent to get started, to test an idea, and that'll be freelance work. That's section one E.
All right, so that's the graduation data. That's really the bulk of what's happening for the graduates for that school. And, so, one of the things CIRR says is you can report a graduation rate, which is the 100% number, and you can report a placement rate. And if you do report a placement rate, you have to use section one, at 180 days. So, I can't go as Fullstack and say, all right, let me have on my website, the combination of one, two, three, at 270 days. It's not CIRR-compliant. I mean, you can do it. But then CIRR, and then the board members of CIRR will give you a call and say hey, what's going on here? You're not compliant with CIRR.
So, and to be honest, most schools are pretty good at adhering to that number. And that number is what I think, is the most truthful to the students about what's happening at that school.
Then, you have a whole section on the median annual base salary of graduates. And, so, this number is calculated dynamically, based off of the range of salaries that that school is obtaining for students. And then gives you kind of a count in different sections. So, here we have under 60, all the way up to over $100,000. And then, section five is what are the most frequent job titles, right? So, here I see front-end engineer, software engineer, fullstack developer, software developer. So, pretty typical titles you would see from companies that are hiring Fullstack Grace Hopper graduates. And then, the section six is something that CIRR added, I think more recently.
What percent of incoming students held a prior computer science degree? I think this is, really, to answer your question, that a lot of students had, was yeah, I feel like bootcamps are getting people jobs, but you know why? That's because all bootcamp grads were CS grads. And that's just not true. We're seeing on the data here, that a very small subsidy of people who had a computer science degree are bootcamp students. And, so, I think that myth is something that should be put to rest, that the only reason bootcamps work is because they're taking CS grads and giving them more up-to-date technology skills.
Bootcamp grads, at least the ones I know, the ones I talk to, they come from all walks of life. To go to a Fullstack program, often times they will have done some studying ahead of time, but they haven't done four years of computer science before they come to Fullstack. And then at the bottom, it just says the pink boxes, if you look online, if you go to CIRR.org, you'll see the pink boxes are the ones the school should be talking about.
And, all right. So that's how you read a report. That's how it works. And then all these reports, like I said earlier, they're audited by an actual auditor. And it's a pretty intensive process.
What Students Should Look Out For
So, some things you should look out for in other metrics, so there are schools that publish metrics that aren't part of CIRR. And things you should look out for are things that I think could create misleading metrics.
And really, the things that can be played with are graduation rate, completion rate, and what it takes to complete. And then also things like not job seeking. So, a lot of those games are about taking people out of the denominator to calculate these numbers. So, if I have 200 graduates and I say, well, you know what? Really about 100 of them weren't job seeking, and then 95 of them got a job. I could say that my rate was 95 percent. Now, CIRR doesn't let you do that. CIRR says look, if 200 people graduated, and you can't say that people were not job seeking to eliminate them from the calculation.
The other thing to look at is graduation rate. So, another thing you can do, is you can say you know what? A lot of our students came here, they paid this tuition, but they didn't really graduate that program. And that could be true. Maybe they left midway. But, what I also see sometimes, is schools will take things like they didn't check in six months from the program, and that means that they didn't really graduate from the program. And I don't think schools are doing anything wrong, so far, but it is a thing that I think can be gamed. So, that's something to be careful about. As you look at bootcamps, you look at outcome metrics, if they aren't CIRR, if they aren't reported in the CIRR style, if they're not CIRR-compliant, then there's other things that you should look out for.
And finally, I want say, what does a CIRR report mean to you, as a student? Really, what I think CIRR has done a very good job of, and really accomplished as a goal, is it keeps the schools very honest, right? As a school operator, I really spend a lot of time and effort keeping in touch with students. I spend a lot of time explaining to them about standards, about how we capture standards, and reasons why they should help us report and have good reporting.Because the good reporting helps us give better outcomes, really, right?
You improve what you measure. And, so, it forces schools to capture data rigorously. And I think it can be, it's helpful for us, as a school, to really know how we're doing.
And it's a dipstick every six months for us to say, how are we doing as a school? And how can we improve? And I think, really, what that means to you as a student, as you're looking at schools, is you should really reach out to a school and say I've looked at your CIRR report, I've looked at your outcomes, and I'm curious how you think me and my background matches with some of the outcomes that you're seeing.
Because this doesn't capture all the different types of backgrounds, and then all the different types of outcomes that happen. But, it does make schools very good at knowing hey, this person, they came from a legal background or a finance background or a design background, and these are the kind of outcomes that we see and we can map, right? So, that's not reported in CIRR, but because we spend so much time capturing this information, and improving this information, we really are able to help students who are interested answer some of those questions.
So, check out CIRR at CIRR.org, C-I-R-R dot O-R-G. I pronounce it sirr, some people pronounce it see-er. I think CIRR is the right way. And again, it stands for Council on Integrity in Results Reporting. And I'm on the board. I've talked to people about CIRR as well. And it's a great organization. I really think people on there are trying to do great work in making sure that students understand what's happening in the boot camp space. All right, thanks so much for your time, and talk to you soon.